2023-01-25T09:41:30.789 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-25T09:41:30.790 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137206 --description rados/standalone/{supported-random-distro$/{centos_8} workloads/scrub} -- /home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137206/orig.config.yaml 2023-01-25T09:41:30.814 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T09:41:30.865 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137206 branch: wip-yuri6-testing-2023-01-24-0746 description: rados/standalone/{supported-random-distro$/{centos_8} workloads/scrub} email: yweinstein@radhat.com first_in_suite: false job_id: '7137206' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-01-24-0746 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri6-testing-2023-01-24-0746 sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2023-01-24-0746 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c targets: smithi101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7tKFB3udt5LaSOmXqs7ZUR4bOoD2pNxvpdxHHjxQkL7wnm8TvW3sKRXeGGly4qxIDL0H/VsD9+3Q8Gplf95jKHQKfqcVcPSRDy7MagdWc1xnBJYZa0hITUGgv9j8HzzNkXvr5Eq0sCtnRl5mOD8AY/lBmdrD8xKOe8FPYXYTqMCoB4bYwDuJHKLL+8Vrzwh35Q7EG1WzXAAyHk9ooGpiXpOeqYxL9NUeGqAuUezHJiSH5hwQigY3Ng+yQJ1CpBMS4J2TwtDy/8IrZl4vOy+VfsJgWs8iPIFZU0NA3x6/DhStDH19ttmSOuHd8VhyUW67OICkrYaVX8klmFQeu1g4nWbKCq3yX+ZnV59W9Y6StXgWuqPYoZUjaiwU8SMR63z9KGthinhnFaeWs4TEiau4AUlMDWWCB/u+C76OCzKwS35fXEz4fgY0ANxe8lPu++Dwgi2MyzuSDg8EcLcTtRJ9HrKgQvCq822DeWRCsB1voz4o+I8dZXW9SL9KaQLGZeHs= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_22:27:30 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-25T09:41:30.866 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa; will attempt to use it 2023-01-25T09:41:30.866 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa/tasks 2023-01-25T09:41:30.867 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-25T09:41:30.868 INFO:teuthology.task.internal:Checking packages... 2023-01-25T09:41:30.894 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '44e4095da12a8269eb6cfe163cbb0d00b1a6412c' 2023-01-25T09:41:30.895 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-25T09:41:30.895 INFO:teuthology.packaging:ref: None 2023-01-25T09:41:30.896 INFO:teuthology.packaging:tag: None 2023-01-25T09:41:30.896 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-01-24-0746 2023-01-25T09:41:30.896 INFO:teuthology.packaging:sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T09:41:30.897 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2023-01-24-0746 2023-01-25T09:41:31.067 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1986.g44e4095d 2023-01-25T09:41:31.068 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-25T09:41:31.076 INFO:teuthology.task.internal:no buildpackages task found 2023-01-25T09:41:31.076 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-25T09:41:31.108 INFO:teuthology.task.internal:Saving configuration 2023-01-25T09:41:31.118 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-25T09:41:31.125 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T09:41:31.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 09:36:36.253812', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPe/eQeVnJumNg4e6xtmVzHfbBYDfARI53JcSnevat7wsn2SHpsoV/y7/CyakMihZ6vKIemhzpawa+NtxzrxSM='} 2023-01-25T09:41:31.147 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-25T09:41:31.154 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-25T09:41:31.154 INFO:teuthology.run_tasks:Running task console_log... 2023-01-25T09:41:31.190 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0d60f8a5e0>, signals=[15]) 2023-01-25T09:41:31.190 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-25T09:41:31.197 INFO:teuthology.task.internal:Opening connections... 2023-01-25T09:41:31.197 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi101.front.sepia.ceph.com 2023-01-25T09:41:31.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T09:41:31.266 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-25T09:41:31.274 DEBUG:teuthology.orchestra.run.smithi101:> uname -m 2023-01-25T09:41:31.292 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64 2023-01-25T09:41:31.293 DEBUG:teuthology.orchestra.run.smithi101:> cat /etc/os-release 2023-01-25T09:41:31.350 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="CentOS Stream" 2023-01-25T09:41:31.350 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="8" 2023-01-25T09:41:31.351 INFO:teuthology.orchestra.run.smithi101.stdout:ID="centos" 2023-01-25T09:41:31.351 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="rhel fedora" 2023-01-25T09:41:31.351 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="8" 2023-01-25T09:41:31.351 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el8" 2023-01-25T09:41:31.351 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-25T09:41:31.352 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31" 2023-01-25T09:41:31.352 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-25T09:41:31.352 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://centos.org/" 2023-01-25T09:41:31.352 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-25T09:41:31.352 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-25T09:41:31.353 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-25T09:41:31.353 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2023-01-25T09:41:31.368 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-25T09:41:31.378 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-25T09:41:31.385 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-25T09:41:31.386 DEBUG:teuthology.orchestra.run.smithi101:> test '!' -e /home/ubuntu/cephtest 2023-01-25T09:41:31.407 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-25T09:41:31.415 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-25T09:41:31.415 DEBUG:teuthology.orchestra.run.smithi101:> test -z $(ls -A /var/lib/ceph) 2023-01-25T09:41:31.481 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-25T09:41:31.512 INFO:teuthology.run_tasks:Running task kernel... 2023-01-25T09:41:31.529 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-25T09:41:31.529 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-25T09:41:31.530 DEBUG:teuthology.orchestra.run.smithi101:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T09:41:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T09:41:31.546 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2023-01-25T09:41:31.603 INFO:teuthology.orchestra.run.smithi101.stdout:4.18.0-408.el8.x86_64 2023-01-25T09:41:31.603 INFO:teuthology.task.kernel:Running kernel on smithi101: 4.18.0-408.el8.x86_64 2023-01-25T09:41:31.604 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum install -y kernel 2023-01-25T09:41:35.206 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - AppStream 40 MB/s | 27 MB 00:00 2023-01-25T09:41:40.898 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - BaseOS 41 MB/s | 27 MB 00:00 2023-01-25T09:41:45.164 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - Extras 160 kB/s | 18 kB 00:00 2023-01-25T09:41:45.533 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - PowerTools 24 MB/s | 5.6 MB 00:00 2023-01-25T09:41:46.732 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-01-25T09:41:48.882 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 13 MB 00:02 2023-01-25T09:41:52.349 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 104 kB/s | 24 kB 00:00 2023-01-25T09:41:55.455 INFO:teuthology.orchestra.run.smithi101.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-25T09:41:55.456 INFO:teuthology.orchestra.run.smithi101.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-25T09:41:55.624 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:41:55.625 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:41:55.625 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:41:55.625 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:41:55.625 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:41:55.626 INFO:teuthology.orchestra.run.smithi101.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-25T09:41:55.626 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-25T09:41:55.626 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-25T09:41:55.626 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:41:55.627 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:41:55.627 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:41:55.627 INFO:teuthology.orchestra.run.smithi101.stdout:Install 3 Packages 2023-01-25T09:41:55.627 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:41:55.628 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 85 M 2023-01-25T09:41:55.628 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 94 M 2023-01-25T09:41:55.628 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:41:55.900 INFO:teuthology.orchestra.run.smithi101.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 34 MB/s | 9.2 MB 00:00 2023-01-25T09:41:56.592 INFO:teuthology.orchestra.run.smithi101.stdout:(2/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 35 MB/s | 34 MB 00:00 2023-01-25T09:41:57.142 INFO:teuthology.orchestra.run.smithi101.stdout:(3/3): kernel-core-4.18.0-448.el8.x86_64.rpm 27 MB/s | 41 MB 00:01 2023-01-25T09:41:57.145 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:41:57.146 INFO:teuthology.orchestra.run.smithi101.stdout:Total 56 MB/s | 85 MB 00:01 2023-01-25T09:41:57.778 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:41:58.156 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:41:58.156 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:42:01.430 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:42:01.449 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:42:04.164 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:42:04.778 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T09:42:08.626 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T09:42:09.053 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T09:42:12.369 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T09:42:12.626 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T09:42:34.436 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-25T09:42:34.462 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T09:42:34.918 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T09:42:34.918 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-25T09:42:34.919 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-25T09:42:35.543 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T09:42:35.543 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:42:35.543 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:42:35.544 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-25T09:42:35.544 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-25T09:42:35.544 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:42:35.544 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:42:36.354 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q kernel --last | head -n 1 2023-01-25T09:42:36.563 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 09:42:12 AM UTC 2023-01-25T09:42:36.565 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T09:42:36.565 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-25T09:42:36.565 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T09:42:36.565 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T09:42:36.566 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-25T09:42:36.566 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi101.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T09:42:36.566 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q kernel --last | head -n 1 2023-01-25T09:42:36.754 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 09:42:12 AM UTC 2023-01-25T09:42:36.756 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T09:42:36.756 DEBUG:teuthology.orchestra.run.smithi101:> sudo rpm -qi grub2-tools 2023-01-25T09:42:36.789 INFO:teuthology.orchestra.run.smithi101.stdout:Name : grub2-tools 2023-01-25T09:42:36.789 INFO:teuthology.orchestra.run.smithi101.stdout:Epoch : 1 2023-01-25T09:42:36.790 INFO:teuthology.orchestra.run.smithi101.stdout:Version : 2.02 2023-01-25T09:42:36.790 INFO:teuthology.orchestra.run.smithi101.stdout:Release : 129.el8 2023-01-25T09:42:36.790 INFO:teuthology.orchestra.run.smithi101.stdout:Architecture: x86_64 2023-01-25T09:42:36.790 INFO:teuthology.orchestra.run.smithi101.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-25T09:42:36.790 INFO:teuthology.orchestra.run.smithi101.stdout:Group : System Environment/Base 2023-01-25T09:42:36.791 INFO:teuthology.orchestra.run.smithi101.stdout:Size : 9265586 2023-01-25T09:42:36.791 INFO:teuthology.orchestra.run.smithi101.stdout:License : GPLv3+ 2023-01-25T09:42:36.791 INFO:teuthology.orchestra.run.smithi101.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-25T09:42:36.791 INFO:teuthology.orchestra.run.smithi101.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-25T09:42:36.791 INFO:teuthology.orchestra.run.smithi101.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-25T09:42:36.792 INFO:teuthology.orchestra.run.smithi101.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-25T09:42:36.792 INFO:teuthology.orchestra.run.smithi101.stdout:Relocations : (not relocatable) 2023-01-25T09:42:36.792 INFO:teuthology.orchestra.run.smithi101.stdout:Packager : CentOS BuildSystem 2023-01-25T09:42:36.792 INFO:teuthology.orchestra.run.smithi101.stdout:Vendor : CentOS 2023-01-25T09:42:36.792 INFO:teuthology.orchestra.run.smithi101.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-25T09:42:36.793 INFO:teuthology.orchestra.run.smithi101.stdout:Summary : Support tools for GRUB. 2023-01-25T09:42:36.793 INFO:teuthology.orchestra.run.smithi101.stdout:Description : 2023-01-25T09:42:36.793 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:42:36.793 INFO:teuthology.orchestra.run.smithi101.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-25T09:42:36.794 INFO:teuthology.orchestra.run.smithi101.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-25T09:42:36.794 INFO:teuthology.orchestra.run.smithi101.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-25T09:42:36.794 INFO:teuthology.orchestra.run.smithi101.stdout:hardware devices. 2023-01-25T09:42:36.794 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:42:36.794 INFO:teuthology.orchestra.run.smithi101.stdout:This subpackage provides tools for support of all platforms. 2023-01-25T09:42:36.795 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-25T09:42:36.796 INFO:teuthology.task.kernel:Updating grub on smithi101 to boot 4.18.0-448.el8.x86_64 2023-01-25T09:42:36.796 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-25T09:42:38.090 INFO:teuthology.orchestra.run.smithi101.stderr:Generating grub configuration file ... 2023-01-25T09:42:39.795 INFO:teuthology.orchestra.run.smithi101.stderr:done 2023-01-25T09:42:39.796 DEBUG:teuthology.orchestra.run.smithi101:> mktemp 2023-01-25T09:42:39.845 INFO:teuthology.orchestra.run.smithi101.stdout:/tmp/tmp.bqbLf9FVCx 2023-01-25T09:42:39.846 DEBUG:teuthology.orchestra.run.smithi101:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bqbLf9FVCx 2023-01-25T09:42:39.879 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0666 /tmp/tmp.bqbLf9FVCx 2023-01-25T09:42:40.042 DEBUG:teuthology.orchestra.remote:smithi101:/tmp/tmp.bqbLf9FVCx is 6KB 2023-01-25T09:42:40.091 DEBUG:teuthology.orchestra.run.smithi101:> rm -fr /tmp/tmp.bqbLf9FVCx 2023-01-25T09:42:40.108 DEBUG:teuthology.orchestra.run.smithi101:> sudo /bin/ls /boot/loader/entries || true 2023-01-25T09:42:40.173 INFO:teuthology.orchestra.run.smithi101.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-25T09:42:40.174 INFO:teuthology.orchestra.run.smithi101.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-25T09:42:40.174 INFO:teuthology.orchestra.run.smithi101.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-25T09:42:40.174 INFO:teuthology.orchestra.run.smithi101.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-25T09:42:40.175 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-25T09:42:40.321 DEBUG:teuthology.orchestra.run.smithi101:> sudo shutdown -r now 2023-01-25T09:43:10.352 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-25T09:43:10.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-01-25T09:43:10.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T09:43:28.872 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.101 2023-01-25T09:43:58.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-01-25T09:43:58.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T09:44:16.135 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-01-25T09:44:16.431 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-01-25T09:44:16.431 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-25T09:44:16.431 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2023-01-25T09:44:16.449 INFO:teuthology.orchestra.run.smithi101.stdout:4.18.0-448.el8.x86_64 2023-01-25T09:44:16.449 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-25T09:44:16.450 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T09:44:16.450 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-25T09:44:17.451 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-25T09:44:17.451 DEBUG:teuthology.orchestra.run.smithi101:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T09:44:17.559 INFO:teuthology.orchestra.run.smithi101.stdout:ttyS1 2023-01-25T09:44:17.588 DEBUG:teuthology.parallel:result is None 2023-01-25T09:44:17.589 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-25T09:44:17.599 INFO:teuthology.task.internal:Creating test directory... 2023-01-25T09:44:17.600 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T09:44:17.620 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-25T09:44:17.627 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-25T09:44:17.639 INFO:teuthology.task.internal:Creating archive directory... 2023-01-25T09:44:17.639 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T09:44:17.687 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-25T09:44:17.693 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-25T09:44:17.694 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T09:44:17.778 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:44:17.788 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:44:17.790 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-25T09:44:17.796 INFO:teuthology.task.internal:Configuring sudo... 2023-01-25T09:44:17.797 DEBUG:teuthology.orchestra.run.smithi101:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T09:44:17.825 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-25T09:44:17.833 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-25T09:44:17.834 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T09:44:17.883 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T09:44:17.966 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T09:44:18.011 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T09:44:18.092 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T09:44:18.138 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-01-25T09:44:18.139 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T09:44:18.212 DEBUG:teuthology.orchestra.run.smithi101:> sudo service rsyslog restart 2023-01-25T09:44:18.312 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-25T09:44:19.277 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-25T09:44:19.291 INFO:teuthology.task.internal:Starting timer... 2023-01-25T09:44:19.292 INFO:teuthology.run_tasks:Running task pcp... 2023-01-25T09:44:19.314 INFO:teuthology.run_tasks:Running task selinux... 2023-01-25T09:44:19.325 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-25T09:44:19.326 DEBUG:teuthology.orchestra.run.smithi101:> sudo service auditd rotate 2023-01-25T09:44:19.421 INFO:teuthology.orchestra.run.smithi101.stdout:Rotating logs: 2023-01-25T09:44:19.423 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-25T09:44:19.423 DEBUG:teuthology.orchestra.run.smithi101:> /usr/sbin/getenforce 2023-01-25T09:44:19.465 INFO:teuthology.orchestra.run.smithi101.stdout:Permissive 2023-01-25T09:44:19.466 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi101.front.sepia.ceph.com': 'permissive'} 2023-01-25T09:44:19.466 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-25T09:44:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T09:44:19.535 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-25T09:44:19.535 DEBUG:teuthology.orchestra.run.smithi101:> sudo /usr/sbin/setenforce permissive 2023-01-25T09:44:19.603 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-25T09:44:19.612 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-25T09:44:19.613 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-25T09:44:19.634 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-25T09:44:19.636 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi101.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-25T09:50:41.986 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi101.front.sepia.ceph.com')] 2023-01-25T09:50:41.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-01-25T09:50:41.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T09:50:42.059 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-01-25T09:50:42.127 INFO:teuthology.run_tasks:Running task clock... 2023-01-25T09:50:42.147 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-25T09:50:42.147 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-25T09:50:42.148 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T09:50:42.200 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-25T09:50:42.214 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-25T09:50:42.289 INFO:teuthology.orchestra.run.smithi101.stderr:sudo: ntpd: command not found 2023-01-25T09:50:42.300 INFO:teuthology.orchestra.run.smithi101.stdout:506 Cannot talk to daemon 2023-01-25T09:50:42.314 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-25T09:50:42.328 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-25T09:50:42.399 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found 2023-01-25T09:50:42.402 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-25T09:50:42.402 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-01-25T09:50:42.402 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T09:50:42.403 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T09:50:42.403 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T09:50:42.403 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T09:50:42.404 INFO:teuthology.run_tasks:Running task install... 2023-01-25T09:50:42.415 DEBUG:teuthology.task.install:project ceph 2023-01-25T09:50:42.416 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'}} 2023-01-25T09:50:42.416 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'} 2023-01-25T09:50:42.416 INFO:teuthology.task.install:Using flavor: default 2023-01-25T09:50:42.422 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-25T09:50:42.422 INFO:teuthology.task.install:extra packages: [] 2023-01-25T09:50:42.423 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2023-01-25T09:50:42.481 INFO:teuthology.orchestra.run.smithi101.stdout:CentOSStream 2023-01-25T09:50:42.482 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T09:50:42.483 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c', 'tag': None, 'wait_for_package': False} 2023-01-25T09:50:42.483 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2023-01-25T09:50:42.559 INFO:teuthology.orchestra.run.smithi101.stdout:CentOSStream 2023-01-25T09:50:42.560 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T09:50:42.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T09:50:42.713 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/ 2023-01-25T09:50:42.713 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1986.g44e4095d 2023-01-25T09:50:42.828 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-25T09:50:42.829 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-01-25T09:50:42.829 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-25T09:50:42.858 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-25T09:50:42.858 DEBUG:teuthology.orchestra.run.smithi101:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-25T09:50:42.925 DEBUG:teuthology.orchestra.run.smithi101:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-25T09:50:43.003 DEBUG:teuthology.orchestra.run.smithi101:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-25T09:50:43.070 INFO:teuthology.orchestra.run.smithi101.stdout:check_obsoletes = 1 2023-01-25T09:50:43.072 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum clean all 2023-01-25T09:50:43.321 INFO:teuthology.orchestra.run.smithi101.stdout:61 files removed 2023-01-25T09:50:43.350 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-radosgw 2023-01-25T09:50:44.048 INFO:teuthology.orchestra.run.smithi101.stdout:ceph packages for x86_64 269 kB/s | 79 kB 00:00 2023-01-25T09:50:44.341 INFO:teuthology.orchestra.run.smithi101.stdout:ceph noarch packages 37 kB/s | 9.3 kB 00:00 2023-01-25T09:50:44.600 INFO:teuthology.orchestra.run.smithi101.stdout:ceph source packages 6.5 kB/s | 1.5 kB 00:00 2023-01-25T09:50:45.310 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - AppStream 40 MB/s | 27 MB 00:00 2023-01-25T09:50:50.878 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-01-25T09:50:55.093 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - Extras 158 kB/s | 18 kB 00:00 2023-01-25T09:50:55.495 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-25T09:50:56.728 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-01-25T09:50:58.812 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 13 MB 00:01 2023-01-25T09:51:02.095 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 466 kB/s | 24 kB 00:00 2023-01-25T09:51:05.534 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:51:05.536 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:51:05.536 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:51:05.537 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:51:05.537 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:51:05.537 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 13 M 2023-01-25T09:51:05.537 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrading: 2023-01-25T09:51:05.538 INFO:teuthology.orchestra.run.smithi101.stdout: librados2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.6 M 2023-01-25T09:51:05.538 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.6 M 2023-01-25T09:51:05.538 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:51:05.538 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 4.8 M 2023-01-25T09:51:05.538 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 21 M 2023-01-25T09:51:05.539 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 24 k 2023-01-25T09:51:05.539 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-25T09:51:05.539 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 767 k 2023-01-25T09:51:05.539 INFO:teuthology.orchestra.run.smithi101.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-25T09:51:05.539 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-25T09:51:05.540 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 474 k 2023-01-25T09:51:05.540 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-25T09:51:05.540 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 6.5 M 2023-01-25T09:51:05.540 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-25T09:51:05.540 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-25T09:51:05.541 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse 2023-01-25T09:51:05.541 INFO:teuthology.orchestra.run.smithi101.stdout: x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 45 k 2023-01-25T09:51:05.541 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 115 k 2023-01-25T09:51:05.541 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 190 k 2023-01-25T09:51:05.542 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-25T09:51:05.542 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 372 k 2023-01-25T09:51:05.542 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 364 k 2023-01-25T09:51:05.542 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 106 k 2023-01-25T09:51:05.542 INFO:teuthology.orchestra.run.smithi101.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-25T09:51:05.543 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:05.543 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:51:05.543 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:51:05.543 INFO:teuthology.orchestra.run.smithi101.stdout:Install 21 Packages 2023-01-25T09:51:05.543 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrade 2 Packages 2023-01-25T09:51:05.544 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:05.544 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 58 M 2023-01-25T09:51:05.545 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:51:06.272 INFO:teuthology.orchestra.run.smithi101.stdout:(1/23): ceph-base-18.0.0-1986.g44e4095d.el8.x86 6.6 MB/s | 4.8 MB 00:00 2023-01-25T09:51:06.330 INFO:teuthology.orchestra.run.smithi101.stdout:(2/23): ceph-selinux-18.0.0-1986.g44e4095d.el8. 418 kB/s | 24 kB 00:00 2023-01-25T09:51:06.489 INFO:teuthology.orchestra.run.smithi101.stdout:(3/23): libcephfs2-18.0.0-1986.g44e4095d.el8.x8 4.7 MB/s | 767 kB 00:00 2023-01-25T09:51:06.662 INFO:teuthology.orchestra.run.smithi101.stdout:(4/23): libradosstriper1-18.0.0-1986.g44e4095d. 2.7 MB/s | 474 kB 00:00 2023-01-25T09:51:07.090 INFO:teuthology.orchestra.run.smithi101.stdout:(5/23): ceph-radosgw-18.0.0-1986.g44e4095d.el8. 8.4 MB/s | 13 MB 00:01 2023-01-25T09:51:07.190 INFO:teuthology.orchestra.run.smithi101.stdout:(6/23): python3-ceph-argparse-18.0.0-1986.g44e4 446 kB/s | 45 kB 00:00 2023-01-25T09:51:07.298 INFO:teuthology.orchestra.run.smithi101.stdout:(7/23): python3-ceph-common-18.0.0-1986.g44e409 1.0 MB/s | 115 kB 00:00 2023-01-25T09:51:07.407 INFO:teuthology.orchestra.run.smithi101.stdout:(8/23): python3-cephfs-18.0.0-1986.g44e4095d.el 1.7 MB/s | 190 kB 00:00 2023-01-25T09:51:07.532 INFO:teuthology.orchestra.run.smithi101.stdout:(9/23): librgw2-18.0.0-1986.g44e4095d.el8.x86_6 7.5 MB/s | 6.5 MB 00:00 2023-01-25T09:51:07.557 INFO:teuthology.orchestra.run.smithi101.stdout:(10/23): python3-rados-18.0.0-1986.g44e4095d.el 2.4 MB/s | 372 kB 00:00 2023-01-25T09:51:07.591 INFO:teuthology.orchestra.run.smithi101.stdout:(11/23): python3-rbd-18.0.0-1986.g44e4095d.el8. 6.1 MB/s | 364 kB 00:00 2023-01-25T09:51:07.792 INFO:teuthology.orchestra.run.smithi101.stdout:(12/23): ceph-common-18.0.0-1986.g44e4095d.el8. 9.5 MB/s | 21 MB 00:02 2023-01-25T09:51:07.817 INFO:teuthology.orchestra.run.smithi101.stdout:(13/23): python3-rgw-18.0.0-1986.g44e4095d.el8. 411 kB/s | 106 kB 00:00 2023-01-25T09:51:07.892 INFO:teuthology.orchestra.run.smithi101.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 MB/s | 354 kB 00:00 2023-01-25T09:51:07.909 INFO:teuthology.orchestra.run.smithi101.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 526 kB/s | 44 kB 00:00 2023-01-25T09:51:07.934 INFO:teuthology.orchestra.run.smithi101.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-01-25T09:51:07.968 INFO:teuthology.orchestra.run.smithi101.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-01-25T09:51:08.068 INFO:teuthology.orchestra.run.smithi101.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 440 kB/s | 59 kB 00:00 2023-01-25T09:51:08.135 INFO:teuthology.orchestra.run.smithi101.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-01-25T09:51:08.369 INFO:teuthology.orchestra.run.smithi101.stdout:(20/23): librados2-18.0.0-1986.g44e4095d.el8.x8 16 MB/s | 3.6 MB 00:00 2023-01-25T09:51:08.436 INFO:teuthology.orchestra.run.smithi101.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 4.7 MB/s | 1.7 MB 00:00 2023-01-25T09:51:08.461 INFO:teuthology.orchestra.run.smithi101.stdout:(22/23): libunwind-1.3.1-3.el8.x86_64.rpm 152 kB/s | 75 kB 00:00 2023-01-25T09:51:08.578 INFO:teuthology.orchestra.run.smithi101.stdout:(23/23): librbd1-18.0.0-1986.g44e4095d.el8.x86_ 17 MB/s | 3.6 MB 00:00 2023-01-25T09:51:08.581 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:51:08.582 INFO:teuthology.orchestra.run.smithi101.stdout:Total 19 MB/s | 58 MB 00:03 2023-01-25T09:51:08.618 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:51:08.662 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:51:08.662 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:51:09.910 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:51:09.913 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:51:11.257 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:51:11.451 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-01-25T09:51:11.670 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-01-25T09:51:11.863 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T09:51:12.265 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T09:51:12.472 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T09:51:12.561 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T09:51:12.819 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 5/25 2023-01-25T09:51:13.005 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 6/25 2023-01-25T09:51:13.401 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 6/25 2023-01-25T09:51:13.588 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T09:51:13.658 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T09:51:13.852 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-25T09:51:13.996 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T09:51:14.087 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T09:51:14.240 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8. 10/25 2023-01-25T09:51:14.465 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 11/25 2023-01-25T09:51:14.698 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 12/25 2023-01-25T09:51:14.838 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 13/25 2023-01-25T09:51:14.908 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 13/25 2023-01-25T09:51:15.144 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-25T09:51:15.931 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-25T09:51:16.105 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 16/25 2023-01-25T09:51:16.172 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 16/25 2023-01-25T09:51:16.345 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 17/25 2023-01-25T09:51:16.538 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-25T09:51:16.763 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x8 19/25 2023-01-25T09:51:20.263 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T09:51:20.488 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T09:51:21.030 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T09:51:21.343 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 21/25 2023-01-25T09:51:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 21/25 2023-01-25T09:51:21.587 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T09:51:40.651 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T09:51:40.652 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /sys 2023-01-25T09:51:40.652 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /proc 2023-01-25T09:51:40.652 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2023-01-25T09:51:40.653 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /run 2023-01-25T09:51:40.653 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /mnt 2023-01-25T09:51:40.653 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /var/tmp 2023-01-25T09:51:40.653 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /home 2023-01-25T09:51:40.654 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /tmp 2023-01-25T09:51:40.654 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2023-01-25T09:51:40.654 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:42.090 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 23/25 2023-01-25T09:51:42.111 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 23/25 2023-01-25T09:51:42.112 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T09:51:42.337 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T09:51:42.337 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T09:51:43.032 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T09:51:43.032 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 1/25 2023-01-25T09:51:43.032 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2/25 2023-01-25T09:51:43.033 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 3/25 2023-01-25T09:51:43.033 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T09:51:43.033 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 5/25 2023-01-25T09:51:43.033 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 6/25 2023-01-25T09:51:43.033 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T09:51:43.034 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8. 8/25 2023-01-25T09:51:43.034 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x8 9/25 2023-01-25T09:51:43.034 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 10/25 2023-01-25T09:51:43.034 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 11/25 2023-01-25T09:51:43.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 12/25 2023-01-25T09:51:43.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 13/25 2023-01-25T09:51:43.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-25T09:51:43.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-25T09:51:43.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-25T09:51:43.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-25T09:51:43.036 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-25T09:51:43.036 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-25T09:51:43.036 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-25T09:51:43.036 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-25T09:51:43.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T09:51:43.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-25T09:51:43.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 24/25 2023-01-25T09:51:43.849 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T09:51:43.850 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:43.850 INFO:teuthology.orchestra.run.smithi101.stdout:Upgraded: 2023-01-25T09:51:43.850 INFO:teuthology.orchestra.run.smithi101.stdout: librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.851 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.852 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:51:43.853 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.853 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.853 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.854 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.854 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-25T09:51:43.855 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.855 INFO:teuthology.orchestra.run.smithi101.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-25T09:51:43.855 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-25T09:51:43.856 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.856 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-25T09:51:43.857 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.857 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-25T09:51:43.857 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-25T09:51:43.858 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.859 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.859 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.859 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-25T09:51:43.860 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.860 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.860 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:43.861 INFO:teuthology.orchestra.run.smithi101.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-25T09:51:43.861 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:43.861 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:51:44.141 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-test 2023-01-25T09:51:44.871 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:00:42 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:51:46.007 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:51:46.007 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:51:46.008 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:51:46.008 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:51:46.008 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:51:46.008 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 45 M 2023-01-25T09:51:46.009 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:51:46.009 INFO:teuthology.orchestra.run.smithi101.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-25T09:51:46.009 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 137 k 2023-01-25T09:51:46.009 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-25T09:51:46.010 INFO:teuthology.orchestra.run.smithi101.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-25T09:51:46.010 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-25T09:51:46.010 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:46.010 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:51:46.010 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:51:46.011 INFO:teuthology.orchestra.run.smithi101.stdout:Install 6 Packages 2023-01-25T09:51:46.011 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:46.011 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 46 M 2023-01-25T09:51:46.012 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 204 M 2023-01-25T09:51:46.012 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:51:46.082 INFO:teuthology.orchestra.run.smithi101.stdout:(1/6): jq-1.6-6.el8.x86_64.rpm 2.7 MB/s | 202 kB 00:00 2023-01-25T09:51:46.107 INFO:teuthology.orchestra.run.smithi101.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.6 MB/s | 187 kB 00:00 2023-01-25T09:51:46.166 INFO:teuthology.orchestra.run.smithi101.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-01-25T09:51:46.191 INFO:teuthology.orchestra.run.smithi101.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 71 kB 00:00 2023-01-25T09:51:46.241 INFO:teuthology.orchestra.run.smithi101.stdout:(5/6): libcephsqlite-18.0.0-1986.g44e4095d.el8. 593 kB/s | 137 kB 00:00 2023-01-25T09:51:48.416 INFO:teuthology.orchestra.run.smithi101.stdout:(6/6): ceph-test-18.0.0-1986.g44e4095d.el8.x86_ 19 MB/s | 45 MB 00:02 2023-01-25T09:51:48.419 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:51:48.420 INFO:teuthology.orchestra.run.smithi101.stdout:Total 19 MB/s | 46 MB 00:02 2023-01-25T09:51:48.462 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:51:48.482 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:51:48.483 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:51:48.777 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:51:48.780 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:51:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:51:49.497 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-01-25T09:51:49.701 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-01-25T09:51:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T09:51:49.953 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T09:51:50.146 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : jq-1.6-6.el8.x86_64 4/6 2023-01-25T09:51:50.325 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 5/6 2023-01-25T09:51:54.696 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 5/6 2023-01-25T09:51:54.963 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 6/6 2023-01-25T09:51:55.242 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 6/6 2023-01-25T09:51:55.242 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 1/6 2023-01-25T09:51:55.243 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 2/6 2023-01-25T09:51:55.243 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : jq-1.6-6.el8.x86_64 3/6 2023-01-25T09:51:55.243 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-01-25T09:51:55.243 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-01-25T09:51:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-01-25T09:51:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:55.800 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:51:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: jq-1.6-6.el8.x86_64 2023-01-25T09:51:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:51:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-25T09:51:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-25T09:51:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-25T09:51:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:55.801 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:51:56.015 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph 2023-01-25T09:51:56.743 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:00:54 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:51:57.885 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:51:57.888 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================= 2023-01-25T09:51:57.888 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:51:57.888 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================= 2023-01-25T09:51:57.889 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:51:57.889 INFO:teuthology.orchestra.run.smithi101.stdout: ceph x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 6.4 k 2023-01-25T09:51:57.889 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:51:57.889 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 2.3 M 2023-01-25T09:51:57.889 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 1.5 M 2023-01-25T09:51:57.890 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 236 k 2023-01-25T09:51:57.890 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.9 M 2023-01-25T09:51:57.890 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 17 M 2023-01-25T09:51:57.890 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-25T09:51:57.890 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-25T09:51:57.891 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-25T09:51:57.891 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-25T09:51:57.891 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-25T09:51:57.891 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-25T09:51:57.891 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-25T09:51:57.892 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-25T09:51:57.892 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-25T09:51:57.892 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-25T09:51:57.892 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-25T09:51:57.893 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-25T09:51:57.893 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-25T09:51:57.893 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-25T09:51:57.893 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-25T09:51:57.893 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-25T09:51:57.894 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-25T09:51:57.894 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-25T09:51:57.894 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-25T09:51:57.894 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-25T09:51:57.894 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-25T09:51:57.894 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-25T09:51:57.895 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-25T09:51:57.895 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-25T09:51:57.895 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-25T09:51:57.895 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-25T09:51:57.895 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:57.896 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:51:57.896 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================= 2023-01-25T09:51:57.896 INFO:teuthology.orchestra.run.smithi101.stdout:Install 32 Packages 2023-01-25T09:51:57.896 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:51:57.897 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 28 M 2023-01-25T09:51:57.897 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 107 M 2023-01-25T09:51:57.897 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:51:58.034 INFO:teuthology.orchestra.run.smithi101.stdout:(1/32): ceph-18.0.0-1986.g44e4095d.el8.x86_64.r 46 kB/s | 6.4 kB 00:00 2023-01-25T09:51:58.226 INFO:teuthology.orchestra.run.smithi101.stdout:(2/32): ceph-mgr-18.0.0-1986.g44e4095d.el8.x86_ 4.6 MB/s | 1.5 MB 00:00 2023-01-25T09:51:58.277 INFO:teuthology.orchestra.run.smithi101.stdout:(3/32): ceph-mds-18.0.0-1986.g44e4095d.el8.x86_ 5.9 MB/s | 2.3 MB 00:00 2023-01-25T09:51:58.360 INFO:teuthology.orchestra.run.smithi101.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1986.g44e4 2.8 MB/s | 236 kB 00:00 2023-01-25T09:51:58.419 INFO:teuthology.orchestra.run.smithi101.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-01-25T09:51:58.444 INFO:teuthology.orchestra.run.smithi101.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2023-01-25T09:51:58.469 INFO:teuthology.orchestra.run.smithi101.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2023-01-25T09:51:58.494 INFO:teuthology.orchestra.run.smithi101.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-25T09:51:58.537 INFO:teuthology.orchestra.run.smithi101.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 11 MB/s | 457 kB 00:00 2023-01-25T09:51:58.563 INFO:teuthology.orchestra.run.smithi101.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.4 MB/s | 237 kB 00:00 2023-01-25T09:51:58.613 INFO:teuthology.orchestra.run.smithi101.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 11 MB/s | 559 kB 00:00 2023-01-25T09:51:58.671 INFO:teuthology.orchestra.run.smithi101.stdout:(12/32): ceph-mon-18.0.0-1986.g44e4095d.el8.x86 6.1 MB/s | 3.9 MB 00:00 2023-01-25T09:51:58.696 INFO:teuthology.orchestra.run.smithi101.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2023-01-25T09:51:58.830 INFO:teuthology.orchestra.run.smithi101.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-01-25T09:51:58.855 INFO:teuthology.orchestra.run.smithi101.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2023-01-25T09:51:58.939 INFO:teuthology.orchestra.run.smithi101.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2023-01-25T09:51:58.964 INFO:teuthology.orchestra.run.smithi101.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-01-25T09:51:58.989 INFO:teuthology.orchestra.run.smithi101.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-01-25T09:51:59.015 INFO:teuthology.orchestra.run.smithi101.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-01-25T09:51:59.190 INFO:teuthology.orchestra.run.smithi101.stdout:(20/32): ceph-osd-18.0.0-1986.g44e4095d.el8.x86 17 MB/s | 17 MB 00:00 2023-01-25T09:51:59.215 INFO:teuthology.orchestra.run.smithi101.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 216 kB/s | 49 kB 00:00 2023-01-25T09:51:59.240 INFO:teuthology.orchestra.run.smithi101.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 260 kB/s | 59 kB 00:00 2023-01-25T09:51:59.291 INFO:teuthology.orchestra.run.smithi101.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 375 kB/s | 19 kB 00:00 2023-01-25T09:51:59.316 INFO:teuthology.orchestra.run.smithi101.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2023-01-25T09:51:59.341 INFO:teuthology.orchestra.run.smithi101.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2023-01-25T09:51:59.367 INFO:teuthology.orchestra.run.smithi101.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2023-01-25T09:51:59.392 INFO:teuthology.orchestra.run.smithi101.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2023-01-25T09:51:59.418 INFO:teuthology.orchestra.run.smithi101.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-01-25T09:51:59.468 INFO:teuthology.orchestra.run.smithi101.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2023-01-25T09:51:59.493 INFO:teuthology.orchestra.run.smithi101.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-01-25T09:51:59.518 INFO:teuthology.orchestra.run.smithi101.stdout:(31/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.6 MB/s | 241 kB 00:00 2023-01-25T09:51:59.543 INFO:teuthology.orchestra.run.smithi101.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-01-25T09:51:59.546 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:51:59.546 INFO:teuthology.orchestra.run.smithi101.stdout:Total 17 MB/s | 28 MB 00:01 2023-01-25T09:51:59.621 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:51:59.658 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:51:59.659 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:00.018 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:00.021 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:01.240 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:01.504 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-25T09:52:01.748 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-25T09:52:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-25T09:52:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-25T09:52:02.494 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-25T09:52:02.666 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-25T09:52:02.901 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-25T09:52:03.062 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-25T09:52:03.277 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-25T09:52:03.548 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-25T09:52:03.794 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-25T09:52:04.034 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-25T09:52:04.424 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-25T09:52:04.701 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-25T09:52:04.914 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-25T09:52:05.163 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-25T09:52:05.424 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T09:52:05.715 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-25T09:52:05.946 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-25T09:52:06.150 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-25T09:52:06.378 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-25T09:52:06.603 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-25T09:52:06.844 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-25T09:52:07.108 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-25T09:52:07.444 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-25T09:52:07.810 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-25T09:52:08.164 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8. 27/32 2023-01-25T09:52:08.384 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 28/32 2023-01-25T09:52:09.828 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 28/32 2023-01-25T09:52:10.048 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 29/32 2023-01-25T09:52:10.438 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 29/32 2023-01-25T09:52:10.623 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 30/32 2023-01-25T09:52:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 30/32 2023-01-25T09:52:10.989 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 31/32 2023-01-25T09:52:11.007 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 31/32 2023-01-25T09:52:11.129 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 32/32 2023-01-25T09:52:11.577 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 32/32 2023-01-25T09:52:11.578 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 1/32 2023-01-25T09:52:11.578 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 2/32 2023-01-25T09:52:11.578 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 3/32 2023-01-25T09:52:11.578 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 4/32 2023-01-25T09:52:11.578 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 5/32 2023-01-25T09:52:11.579 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8. 6/32 2023-01-25T09:52:11.579 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-25T09:52:11.579 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-25T09:52:11.579 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-25T09:52:11.579 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-25T09:52:11.579 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-25T09:52:11.580 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-25T09:52:11.580 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-25T09:52:11.580 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-25T09:52:11.580 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-25T09:52:11.580 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-25T09:52:11.581 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T09:52:11.581 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-25T09:52:11.581 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-25T09:52:11.581 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-25T09:52:11.581 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-25T09:52:11.582 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-25T09:52:11.582 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-25T09:52:11.582 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-25T09:52:11.582 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-25T09:52:11.582 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-25T09:52:11.583 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-25T09:52:11.583 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-25T09:52:11.584 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-25T09:52:11.584 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-25T09:52:11.584 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-25T09:52:12.534 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-25T09:52:12.534 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:12.534 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:12.534 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:52:12.535 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:52:12.535 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:52:12.535 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:12.535 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:52:12.535 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:52:12.536 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-25T09:52:12.536 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-25T09:52:12.536 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-25T09:52:12.536 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-25T09:52:12.537 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-25T09:52:12.537 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-25T09:52:12.537 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-25T09:52:12.537 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-25T09:52:12.537 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-25T09:52:12.538 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-25T09:52:12.538 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-25T09:52:12.538 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-25T09:52:12.538 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-25T09:52:12.538 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend-2.6-1.el8.noarch 2023-01-25T09:52:12.539 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-25T09:52:12.539 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-25T09:52:12.539 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-25T09:52:12.539 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-25T09:52:12.539 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-25T09:52:12.540 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-25T09:52:12.540 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-25T09:52:12.540 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-25T09:52:12.540 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-25T09:52:12.540 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-25T09:52:12.541 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-25T09:52:12.541 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-25T09:52:12.541 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:12.541 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:12.723 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-base 2023-01-25T09:52:13.440 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:11 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:14.475 INFO:teuthology.orchestra.run.smithi101.stdout:Package ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:52:14.573 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:14.574 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:52:14.574 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:14.678 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install cephadm 2023-01-25T09:52:15.391 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:13 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:16.524 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:16.525 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:16.526 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:52:16.526 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:16.526 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:16.526 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 82 k 2023-01-25T09:52:16.527 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:16.527 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:16.527 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:16.527 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:52:16.527 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:16.528 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 82 k 2023-01-25T09:52:16.528 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 374 k 2023-01-25T09:52:16.528 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:16.705 INFO:teuthology.orchestra.run.smithi101.stdout:cephadm-18.0.0-1986.g44e4095d.el8.noarch.rpm 460 kB/s | 82 kB 00:00 2023-01-25T09:52:16.706 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:16.706 INFO:teuthology.orchestra.run.smithi101.stdout:Total 457 kB/s | 82 kB 00:00 2023-01-25T09:52:16.706 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:16.714 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:16.714 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:16.715 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:16.719 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:16.767 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:18.325 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:52:18.480 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:52:18.715 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:52:19.151 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:52:19.152 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:19.152 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:19.152 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:19.152 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:19.153 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:19.337 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-immutable-object-cache 2023-01-25T09:52:20.065 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:18 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:21.208 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:21.209 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:21.209 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-01-25T09:52:21.209 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:21.210 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:21.210 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 165 k 2023-01-25T09:52:21.210 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:21.210 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:21.210 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:21.211 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:52:21.211 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:21.211 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 165 k 2023-01-25T09:52:21.211 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 503 k 2023-01-25T09:52:21.211 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:21.727 INFO:teuthology.orchestra.run.smithi101.stdout:ceph-immutable-object-cache-18.0.0-1986.g44e409 320 kB/s | 165 kB 00:00 2023-01-25T09:52:21.727 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:21.728 INFO:teuthology.orchestra.run.smithi101.stdout:Total 319 kB/s | 165 kB 00:00 2023-01-25T09:52:21.728 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:21.740 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:21.741 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:21.795 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:21.799 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:21.932 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:22.113 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T09:52:22.552 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T09:52:23.038 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T09:52:23.039 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:23.039 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:23.039 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:52:23.039 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:23.039 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:23.235 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr 2023-01-25T09:52:23.965 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:21 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:25.003 INFO:teuthology.orchestra.run.smithi101.stdout:Package ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:52:25.104 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:25.105 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:52:25.105 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:25.199 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-dashboard 2023-01-25T09:52:25.914 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:23 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:27.028 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:27.030 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:27.030 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:52:27.031 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:27.031 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:27.031 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 1.7 M 2023-01-25T09:52:27.031 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:52:27.032 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 23 k 2023-01-25T09:52:27.032 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 15 k 2023-01-25T09:52:27.032 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-25T09:52:27.032 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-25T09:52:27.032 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-25T09:52:27.033 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:27.033 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:27.033 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:27.033 INFO:teuthology.orchestra.run.smithi101.stdout:Install 6 Packages 2023-01-25T09:52:27.033 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:27.034 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 2.0 M 2023-01-25T09:52:27.034 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 8.2 M 2023-01-25T09:52:27.034 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:27.649 INFO:teuthology.orchestra.run.smithi101.stdout:(1/6): ceph-prometheus-alerts-18.0.0-1986.g44e4 24 kB/s | 15 kB 00:00 2023-01-25T09:52:27.933 INFO:teuthology.orchestra.run.smithi101.stdout:(2/6): ceph-grafana-dashboards-18.0.0-1986.g44e 26 kB/s | 23 kB 00:00 2023-01-25T09:52:27.992 INFO:teuthology.orchestra.run.smithi101.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 125 kB/s | 43 kB 00:00 2023-01-25T09:52:28.034 INFO:teuthology.orchestra.run.smithi101.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2023-01-25T09:52:28.084 INFO:teuthology.orchestra.run.smithi101.stdout:(5/6): ceph-mgr-dashboard-18.0.0-1986.g44e4095d 1.6 MB/s | 1.7 MB 00:01 2023-01-25T09:52:28.176 INFO:teuthology.orchestra.run.smithi101.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-01-25T09:52:28.178 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:28.179 INFO:teuthology.orchestra.run.smithi101.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-01-25T09:52:28.190 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:28.201 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:28.201 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:28.289 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:28.293 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:28.583 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-25T09:52:29.020 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-25T09:52:29.189 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-25T09:52:29.301 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.n 4/6 2023-01-25T09:52:29.691 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8. 5/6 2023-01-25T09:52:29.887 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 6/6 2023-01-25T09:52:30.190 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 6/6 2023-01-25T09:52:30.191 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8. 1/6 2023-01-25T09:52:30.191 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 2/6 2023-01-25T09:52:30.191 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.n 3/6 2023-01-25T09:52:30.191 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-25T09:52:30.192 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-25T09:52:30.695 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-25T09:52:30.696 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:30.696 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:30.696 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:30.696 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:30.697 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:30.697 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-25T09:52:30.697 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-25T09:52:30.697 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-25T09:52:30.697 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:30.697 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:30.819 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-25T09:52:31.537 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:29 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:32.678 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:32.679 INFO:teuthology.orchestra.run.smithi101.stdout:===================================================================================================== 2023-01-25T09:52:32.679 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:52:32.680 INFO:teuthology.orchestra.run.smithi101.stdout:===================================================================================================== 2023-01-25T09:52:32.680 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:32.680 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 7.4 M 2023-01-25T09:52:32.680 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:52:32.680 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-25T09:52:32.681 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-25T09:52:32.681 INFO:teuthology.orchestra.run.smithi101.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-25T09:52:32.681 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-25T09:52:32.681 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-25T09:52:32.682 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-25T09:52:32.682 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-25T09:52:32.682 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:32.682 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:32.682 INFO:teuthology.orchestra.run.smithi101.stdout:===================================================================================================== 2023-01-25T09:52:32.683 INFO:teuthology.orchestra.run.smithi101.stdout:Install 8 Packages 2023-01-25T09:52:32.683 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:32.683 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 35 M 2023-01-25T09:52:32.684 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 210 M 2023-01-25T09:52:32.684 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:32.856 INFO:teuthology.orchestra.run.smithi101.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-01-25T09:52:33.023 INFO:teuthology.orchestra.run.smithi101.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 22 MB/s | 3.7 MB 00:00 2023-01-25T09:52:33.049 INFO:teuthology.orchestra.run.smithi101.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 8.8 MB/s | 225 kB 00:00 2023-01-25T09:52:33.107 INFO:teuthology.orchestra.run.smithi101.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2023-01-25T09:52:33.158 INFO:teuthology.orchestra.run.smithi101.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 13 MB/s | 645 kB 00:00 2023-01-25T09:52:33.191 INFO:teuthology.orchestra.run.smithi101.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-01-25T09:52:33.300 INFO:teuthology.orchestra.run.smithi101.stdout:(7/8): ceph-mgr-diskprediction-local-18.0.0-198 12 MB/s | 7.4 MB 00:00 2023-01-25T09:52:33.500 INFO:teuthology.orchestra.run.smithi101.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 30 MB/s | 14 MB 00:00 2023-01-25T09:52:33.502 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:33.503 INFO:teuthology.orchestra.run.smithi101.stdout:Total 43 MB/s | 35 MB 00:00 2023-01-25T09:52:33.735 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:33.753 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:33.753 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:34.147 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:34.150 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:34.713 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:34.869 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T09:52:35.003 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T09:52:35.152 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T09:52:35.747 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T09:52:35.919 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T09:52:36.540 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T09:52:36.702 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T09:52:37.392 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T09:52:37.747 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T09:52:37.977 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T09:52:39.818 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T09:52:41.087 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-25T09:52:41.301 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 8/8 2023-01-25T09:52:41.627 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 8/8 2023-01-25T09:52:41.628 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 1/8 2023-01-25T09:52:41.628 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-25T09:52:41.629 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-25T09:52:41.629 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-25T09:52:41.629 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T09:52:41.630 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-25T09:52:41.630 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-25T09:52:42.151 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-25T09:52:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:42.152 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-25T09:52:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-25T09:52:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-25T09:52:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-25T09:52:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-25T09:52:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-25T09:52:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-25T09:52:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:42.154 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:42.276 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-rook 2023-01-25T09:52:42.997 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:40 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:44.126 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:44.127 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================== 2023-01-25T09:52:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:52:44.128 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================== 2023-01-25T09:52:44.128 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 47 k 2023-01-25T09:52:44.129 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:52:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-25T09:52:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-25T09:52:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-25T09:52:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-25T09:52:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-25T09:52:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-25T09:52:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-25T09:52:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-25T09:52:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-25T09:52:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-25T09:52:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-25T09:52:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-25T09:52:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:44.131 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:44.132 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================== 2023-01-25T09:52:44.132 INFO:teuthology.orchestra.run.smithi101.stdout:Install 13 Packages 2023-01-25T09:52:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:44.133 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 1.9 M 2023-01-25T09:52:44.133 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 25 M 2023-01-25T09:52:44.133 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:44.190 INFO:teuthology.orchestra.run.smithi101.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 450 kB/s | 27 kB 00:00 2023-01-25T09:52:44.241 INFO:teuthology.orchestra.run.smithi101.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-01-25T09:52:44.283 INFO:teuthology.orchestra.run.smithi101.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.6 MB/s | 110 kB 00:00 2023-01-25T09:52:44.308 INFO:teuthology.orchestra.run.smithi101.stdout:(4/13): ceph-mgr-rook-18.0.0-1986.g44e4095d.el8 268 kB/s | 47 kB 00:00 2023-01-25T09:52:44.341 INFO:teuthology.orchestra.run.smithi101.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.6 MB/s | 155 kB 00:00 2023-01-25T09:52:44.400 INFO:teuthology.orchestra.run.smithi101.stdout:(6/13): python3-jsonpointer-1.10-11.el8.noarch. 74 kB/s | 20 kB 00:00 2023-01-25T09:52:44.426 INFO:teuthology.orchestra.run.smithi101.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 363 kB/s | 43 kB 00:00 2023-01-25T09:52:44.526 INFO:teuthology.orchestra.run.smithi101.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2023-01-25T09:52:44.551 INFO:teuthology.orchestra.run.smithi101.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 157 kB/s | 33 kB 00:00 2023-01-25T09:52:44.576 INFO:teuthology.orchestra.run.smithi101.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 540 kB/s | 81 kB 00:00 2023-01-25T09:52:44.627 INFO:teuthology.orchestra.run.smithi101.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-01-25T09:52:44.652 INFO:teuthology.orchestra.run.smithi101.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 613 kB/s | 61 kB 00:00 2023-01-25T09:52:44.769 INFO:teuthology.orchestra.run.smithi101.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.8 MB/s | 1.2 MB 00:00 2023-01-25T09:52:44.771 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:44.771 INFO:teuthology.orchestra.run.smithi101.stdout:Total 3.0 MB/s | 1.9 MB 00:00 2023-01-25T09:52:44.824 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:44.839 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:44.839 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:44.936 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:44.940 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:45.438 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:45.711 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-25T09:52:45.901 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-25T09:52:46.136 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-25T09:52:46.374 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-25T09:52:46.573 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-25T09:52:46.805 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-25T09:52:47.112 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-25T09:52:47.355 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-25T09:52:48.412 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-25T09:52:48.740 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-25T09:52:49.750 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-25T09:52:49.961 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-25T09:52:50.117 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 13/13 2023-01-25T09:52:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 13/13 2023-01-25T09:52:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 1/13 2023-01-25T09:52:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-25T09:52:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-25T09:52:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-25T09:52:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-25T09:52:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-25T09:52:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-25T09:52:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-25T09:52:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-25T09:52:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-25T09:52:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-25T09:52:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-25T09:52:50.999 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-25T09:52:50.999 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:50.999 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:51.000 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:51.000 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-25T09:52:51.000 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-25T09:52:51.000 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-25T09:52:51.000 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-25T09:52:51.001 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-25T09:52:51.001 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-25T09:52:51.001 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-25T09:52:51.001 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-25T09:52:51.002 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-25T09:52:51.002 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-25T09:52:51.002 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-25T09:52:51.002 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-25T09:52:51.002 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:51.003 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:51.126 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-cephadm 2023-01-25T09:52:51.857 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:49 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:53.011 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:53.012 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:53.012 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:52:53.012 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:53.013 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:53.013 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 120 k 2023-01-25T09:52:53.013 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:52:53.013 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-25T09:52:53.013 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-25T09:52:53.014 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-25T09:52:53.014 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-25T09:52:53.014 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:53.014 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:53.014 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:53.015 INFO:teuthology.orchestra.run.smithi101.stdout:Install 5 Packages 2023-01-25T09:52:53.015 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:53.015 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 6.0 M 2023-01-25T09:52:53.016 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 26 M 2023-01-25T09:52:53.016 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:53.100 INFO:teuthology.orchestra.run.smithi101.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.1 MB/s | 538 kB 00:00 2023-01-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi101.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-01-25T09:52:53.376 INFO:teuthology.orchestra.run.smithi101.stdout:(3/5): ceph-mgr-cephadm-18.0.0-1986.g44e4095d.e 332 kB/s | 120 kB 00:00 2023-01-25T09:52:53.401 INFO:teuthology.orchestra.run.smithi101.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 301 kB/s | 60 kB 00:00 2023-01-25T09:52:53.501 INFO:teuthology.orchestra.run.smithi101.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2023-01-25T09:52:53.503 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:53.503 INFO:teuthology.orchestra.run.smithi101.stdout:Total 12 MB/s | 6.0 MB 00:00 2023-01-25T09:52:53.555 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:53.566 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:53.566 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:53.708 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:53.711 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:54.007 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:54.304 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-25T09:52:55.249 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-25T09:52:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-25T09:52:55.927 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-25T09:52:56.108 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 5/5 2023-01-25T09:52:56.403 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 5/5 2023-01-25T09:52:56.404 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/5 2023-01-25T09:52:56.404 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-25T09:52:56.404 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-25T09:52:56.404 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-25T09:52:56.904 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-25T09:52:56.904 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:56.904 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:52:56.904 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:52:56.905 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-25T09:52:56.905 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-25T09:52:56.905 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-25T09:52:56.905 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-25T09:52:56.905 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:56.906 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:52:57.035 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-fuse 2023-01-25T09:52:57.752 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:55 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:52:58.897 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:52:58.897 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:58.898 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:52:58.898 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:58.898 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:52:58.898 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 845 k 2023-01-25T09:52:58.898 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-01-25T09:52:58.899 INFO:teuthology.orchestra.run.smithi101.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-25T09:52:58.899 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:58.899 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:52:58.899 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:52:58.899 INFO:teuthology.orchestra.run.smithi101.stdout:Install 2 Packages 2023-01-25T09:52:58.900 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:52:58.900 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 928 k 2023-01-25T09:52:58.900 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 2.9 M 2023-01-25T09:52:58.901 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:52:58.967 INFO:teuthology.orchestra.run.smithi101.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-01-25T09:52:59.292 INFO:teuthology.orchestra.run.smithi101.stdout:(2/2): ceph-fuse-18.0.0-1986.g44e4095d.el8.x86_ 2.1 MB/s | 845 kB 00:00 2023-01-25T09:52:59.293 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:52:59.293 INFO:teuthology.orchestra.run.smithi101.stdout:Total 2.3 MB/s | 928 kB 00:00 2023-01-25T09:52:59.303 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:52:59.325 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:52:59.325 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:52:59.417 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:52:59.421 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:52:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:52:59.864 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-25T09:53:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2/2 2023-01-25T09:53:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2/2 2023-01-25T09:53:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/2 2023-01-25T09:53:01.134 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-25T09:53:01.135 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:01.135 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:01.135 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-25T09:53:01.135 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:01.135 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:01.347 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-volume 2023-01-25T09:53:02.079 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:00 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:03.221 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:03.222 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:03.222 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-01-25T09:53:03.222 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:03.222 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:53:03.223 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 263 k 2023-01-25T09:53:03.223 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:03.223 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:53:03.223 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:03.223 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:53:03.224 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:03.224 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 263 k 2023-01-25T09:53:03.224 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 1.3 M 2023-01-25T09:53:03.224 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:53:03.471 INFO:teuthology.orchestra.run.smithi101.stdout:ceph-volume-18.0.0-1986.g44e4095d.el8.noarch.rp 1.0 MB/s | 263 kB 00:00 2023-01-25T09:53:03.472 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:53:03.472 INFO:teuthology.orchestra.run.smithi101.stdout:Total 1.0 MB/s | 263 kB 00:00 2023-01-25T09:53:03.472 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:53:03.482 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:53:03.482 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:53:03.573 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:53:03.577 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:53:03.762 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:53:04.000 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:53:04.438 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:53:04.904 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T09:53:04.905 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:04.905 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:04.905 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T09:53:04.906 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:04.906 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:05.084 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install librados-devel 2023-01-25T09:53:05.814 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:03 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:06.962 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:06.963 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:06.963 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-01-25T09:53:06.964 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:06.964 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:53:06.964 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 139 k 2023-01-25T09:53:06.964 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:06.964 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:53:06.965 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:06.965 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:53:06.965 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:06.965 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 139 k 2023-01-25T09:53:06.965 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 512 k 2023-01-25T09:53:06.966 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:53:07.158 INFO:teuthology.orchestra.run.smithi101.stdout:librados-devel-18.0.0-1986.g44e4095d.el8.x86_64 720 kB/s | 139 kB 00:00 2023-01-25T09:53:07.158 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:53:07.159 INFO:teuthology.orchestra.run.smithi101.stdout:Total 715 kB/s | 139 kB 00:00 2023-01-25T09:53:07.159 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:53:07.163 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:53:07.164 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:53:07.216 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:53:07.219 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:53:07.359 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:53:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:07.833 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:08.296 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:53:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:08.297 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:08.517 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install libcephfs2 2023-01-25T09:53:09.244 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:07 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:10.277 INFO:teuthology.orchestra.run.smithi101.stdout:Package libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:10.376 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:10.377 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:10.377 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:10.469 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install libcephfs-devel 2023-01-25T09:53:11.188 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:09 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:12.326 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:12.326 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:12.326 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-01-25T09:53:12.327 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:12.327 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:53:12.327 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 30 k 2023-01-25T09:53:12.327 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:12.327 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:53:12.328 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:12.328 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:53:12.328 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:12.329 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 30 k 2023-01-25T09:53:12.329 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 137 k 2023-01-25T09:53:12.329 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:53:12.813 INFO:teuthology.orchestra.run.smithi101.stdout:libcephfs-devel-18.0.0-1986.g44e4095d.el8.x86_6 63 kB/s | 30 kB 00:00 2023-01-25T09:53:12.814 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:53:12.814 INFO:teuthology.orchestra.run.smithi101.stdout:Total 62 kB/s | 30 kB 00:00 2023-01-25T09:53:12.815 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:53:12.818 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:53:12.818 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:53:12.839 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:53:12.843 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:53:12.908 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:53:13.120 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:13.394 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:13.844 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:13.844 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:13.845 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:13.845 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:53:13.845 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:13.845 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:13.966 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install librados2 2023-01-25T09:53:14.695 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:12 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:15.737 INFO:teuthology.orchestra.run.smithi101.stdout:Package librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:15.839 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:15.840 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:15.841 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:15.941 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install librbd1 2023-01-25T09:53:16.658 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:14 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:17.698 INFO:teuthology.orchestra.run.smithi101.stdout:Package librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:17.797 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:17.798 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:17.798 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:17.888 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-rados 2023-01-25T09:53:18.606 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:16 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:19.644 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:19.745 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:19.746 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:19.746 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:19.844 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-rgw 2023-01-25T09:53:20.561 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:18 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:21.615 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:21.717 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:21.718 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:21.718 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:21.822 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-cephfs 2023-01-25T09:53:22.536 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:20 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:23.592 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:23.692 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:23.694 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:23.694 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:23.793 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-rbd 2023-01-25T09:53:24.511 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:22 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:25.556 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T09:53:25.658 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:25.659 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-01-25T09:53:25.659 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:25.760 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install rbd-fuse 2023-01-25T09:53:26.491 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:24 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:27.634 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:27.635 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:27.635 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-01-25T09:53:27.635 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:27.635 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:53:27.635 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 94 k 2023-01-25T09:53:27.636 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:27.636 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:53:27.636 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:27.636 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:53:27.636 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:27.637 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 94 k 2023-01-25T09:53:27.637 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 280 k 2023-01-25T09:53:27.637 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:53:27.843 INFO:teuthology.orchestra.run.smithi101.stdout:rbd-fuse-18.0.0-1986.g44e4095d.el8.x86_64.rpm 454 kB/s | 94 kB 00:00 2023-01-25T09:53:27.844 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:53:27.844 INFO:teuthology.orchestra.run.smithi101.stdout:Total 452 kB/s | 94 kB 00:00 2023-01-25T09:53:27.845 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:53:27.854 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:53:27.855 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:53:27.905 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:53:27.909 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:53:28.053 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:53:28.496 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:28.792 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:29.340 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:29.340 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:29.341 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:29.341 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:53:29.341 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:29.341 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:29.542 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install rbd-mirror 2023-01-25T09:53:30.275 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:28 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:31.423 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:31.424 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:31.424 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-01-25T09:53:31.424 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:31.424 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:53:31.424 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.5 M 2023-01-25T09:53:31.425 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:31.425 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:53:31.425 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:31.425 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:53:31.425 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:31.426 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 3.5 M 2023-01-25T09:53:31.426 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 16 M 2023-01-25T09:53:31.427 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:53:32.223 INFO:teuthology.orchestra.run.smithi101.stdout:rbd-mirror-18.0.0-1986.g44e4095d.el8.x86_64.rpm 4.3 MB/s | 3.5 MB 00:00 2023-01-25T09:53:32.223 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:53:32.224 INFO:teuthology.orchestra.run.smithi101.stdout:Total 4.3 MB/s | 3.5 MB 00:00 2023-01-25T09:53:32.224 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:53:32.236 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:53:32.236 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:53:32.300 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:53:32.304 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:53:32.753 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:53:32.920 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:33.364 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:33.821 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:33.821 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:33.821 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:33.822 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:53:33.822 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:33.822 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:34.069 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install rbd-nbd 2023-01-25T09:53:34.797 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:32 ago on Wed 25 Jan 2023 09:51:02 AM UTC. 2023-01-25T09:53:35.938 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-01-25T09:53:35.939 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:35.939 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-01-25T09:53:35.940 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:35.940 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-01-25T09:53:35.940 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 180 k 2023-01-25T09:53:35.940 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:35.941 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-01-25T09:53:35.941 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-01-25T09:53:35.941 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-01-25T09:53:35.941 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:35.941 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 180 k 2023-01-25T09:53:35.941 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 568 k 2023-01-25T09:53:35.942 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-01-25T09:53:36.128 INFO:teuthology.orchestra.run.smithi101.stdout:rbd-nbd-18.0.0-1986.g44e4095d.el8.x86_64.rpm 965 kB/s | 180 kB 00:00 2023-01-25T09:53:36.128 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-01-25T09:53:36.128 INFO:teuthology.orchestra.run.smithi101.stdout:Total 959 kB/s | 180 kB 00:00 2023-01-25T09:53:36.129 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-01-25T09:53:36.138 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-01-25T09:53:36.138 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-01-25T09:53:36.190 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-01-25T09:53:36.193 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-01-25T09:53:36.321 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-01-25T09:53:36.475 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:36.777 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:37.239 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T09:53:37.240 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:37.240 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-01-25T09:53:37.240 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T09:53:37.240 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-01-25T09:53:37.240 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-01-25T09:53:37.422 DEBUG:teuthology.parallel:result is None 2023-01-25T09:53:37.422 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2023-01-25T09:53:37.459 INFO:teuthology.orchestra.run.smithi101.stdout:CentOSStream 2023-01-25T09:53:37.461 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T09:53:37.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T09:53:37.663 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-25T09:53:37.685 INFO:teuthology.orchestra.run.smithi101.stdout:18.0.0-1986.g44e4095d.el8 2023-01-25T09:53:37.686 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1986.g44e4095d.el8 2023-01-25T09:53:37.687 INFO:teuthology.task.install:The correct ceph version 18.0.0-1986.g44e4095d is installed. 2023-01-25T09:53:37.687 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-25T09:53:37.687 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-01-25T09:53:37.688 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-25T09:53:37.752 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-25T09:53:37.752 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-01-25T09:53:37.753 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/daemon-helper 2023-01-25T09:53:37.819 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-25T09:53:37.886 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-25T09:53:37.886 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-01-25T09:53:37.887 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-25T09:53:37.953 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-25T09:53:38.020 INFO:teuthology.run_tasks:Running task workunit... 2023-01-25T09:53:38.033 INFO:tasks.workunit:Pulling workunits from ref 44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T09:53:38.034 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-25T09:53:38.034 INFO:tasks.workunit:timeout=3h 2023-01-25T09:53:38.035 INFO:tasks.workunit:cleanup=True 2023-01-25T09:53:38.035 DEBUG:teuthology.orchestra.run.smithi101:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-25T09:53:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T09:53:38.078 INFO:teuthology.orchestra.run.smithi101.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-25T09:53:38.079 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-25T09:53:38.136 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-25T09:53:38.136 DEBUG:teuthology.orchestra.run.smithi101:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-25T09:53:38.194 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T09:53:38.253 INFO:tasks.workunit.client.0.smithi101.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-25T09:54:17.011 INFO:tasks.workunit.client.0.smithi101.stderr:Note: switching to '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'. 2023-01-25T09:54:17.012 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.012 INFO:tasks.workunit.client.0.smithi101.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-25T09:54:17.012 INFO:tasks.workunit.client.0.smithi101.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-25T09:54:17.012 INFO:tasks.workunit.client.0.smithi101.stderr:state without impacting any branches by switching back to a branch. 2023-01-25T09:54:17.012 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.013 INFO:tasks.workunit.client.0.smithi101.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-25T09:54:17.013 INFO:tasks.workunit.client.0.smithi101.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-25T09:54:17.013 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.013 INFO:tasks.workunit.client.0.smithi101.stderr: git switch -c 2023-01-25T09:54:17.013 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.014 INFO:tasks.workunit.client.0.smithi101.stderr:Or undo this operation with: 2023-01-25T09:54:17.014 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.014 INFO:tasks.workunit.client.0.smithi101.stderr: git switch - 2023-01-25T09:54:17.014 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.014 INFO:tasks.workunit.client.0.smithi101.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-25T09:54:17.014 INFO:tasks.workunit.client.0.smithi101.stderr: 2023-01-25T09:54:17.015 INFO:tasks.workunit.client.0.smithi101.stderr:HEAD is now at 44e4095da12 Merge branch 'wip-nitzan-filestore-removal' of https://github.com/NitzanMordhai/ceph into wip-yuri6-testing-2023-01-24-0746 2023-01-25T09:54:17.016 DEBUG:teuthology.orchestra.run.smithi101:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-25T09:54:17.075 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-01-25T09:54:17.076 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-25T09:54:17.133 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2023-01-25T09:54:17.134 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2023-01-25T09:54:17.134 DEBUG:teuthology.orchestra.run.smithi101:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=44e4095da12a8269eb6cfe163cbb0d00b1a6412c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2023-01-25T09:54:17.216 INFO:tasks.workunit.client.0.smithi101.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-01-25T09:54:17.224 INFO:tasks.workunit.client.0.smithi101.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-01-25T09:54:17.224 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-01-25T09:54:17.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-01-25T09:54:17.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: export PYTHONWARNINGS=ignore 2023-01-25T09:54:17.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: PYTHONWARNINGS=ignore 2023-01-25T09:54:17.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export CEPH_CONF=/dev/null 2023-01-25T09:54:17.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: CEPH_CONF=/dev/null 2023-01-25T09:54:17.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: unset CEPH_ARGS 2023-01-25T09:54:17.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: local code 2023-01-25T09:54:17.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: main: run td/osd-mapper 2023-01-25T09:54:17.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2023-01-25T09:54:17.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2023-01-25T09:54:17.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2023-01-25T09:54:17.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2023-01-25T09:54:17.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2023-01-25T09:54:17.227 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2023-01-25T09:54:17.229 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none ' 2023-01-25T09:54:17.230 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2023-01-25T09:54:17.230 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-01-25T09:54:17.230 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2023-01-25T09:54:17.231 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-01-25T09:54:17.232 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2023-01-25T09:54:17.232 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2023-01-25T09:54:17.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2023-01-25T09:54:17.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2023-01-25T09:54:17.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2023-01-25T09:54:17.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-01-25T09:54:17.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T09:54:17.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-01-25T09:54:17.234 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T09:54:17.234 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T09:54:17.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T09:54:17.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T09:54:17.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T09:54:17.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T09:54:17.236 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T09:54:17.236 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T09:54:17.236 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T09:54:17.236 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T09:54:17.236 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T09:54:17.237 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T09:54:17.237 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:54:17.238 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T09:54:17.238 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T09:54:17.238 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:54:17.238 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T09:54:17.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T09:54:17.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-01-25T09:54:17.240 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T09:54:17.240 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:54:17.241 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:54:17.241 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99476 2023-01-25T09:54:17.241 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T09:54:17.242 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T09:54:17.242 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2023-01-25T09:54:17.243 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T09:54:17.243 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:54:17.243 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:54:17.243 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.99476 2023-01-25T09:54:17.245 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T09:54:17.245 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-01-25T09:54:17.245 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-01-25T09:54:17.245 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T09:54:17.245 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2023-01-25T09:54:17.246 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2023-01-25T09:54:17.246 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2023-01-25T09:54:17.246 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']="3" ['pgs_in_pool']="4" ['pool_name']="test") 2023-01-25T09:54:17.246 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2023-01-25T09:54:17.246 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2023-01-25T09:54:17.247 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2023-01-25T09:54:17.247 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2023-01-25T09:54:17.247 INFO:tasks.workunit.client.0.smithi101.stdout:Dir: td/osd-mapper 2023-01-25T09:54:17.248 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2023-01-25T09:54:17.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2023-01-25T09:54:17.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2023-01-25T09:54:17.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2023-01-25T09:54:17.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2023-01-25T09:54:17.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2023-01-25T09:54:17.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2023-01-25T09:54:17.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2023-01-25T09:54:17.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2023-01-25T09:54:17.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2023-01-25T09:54:17.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2023-01-25T09:54:19.043 INFO:tasks.workunit.client.0.smithi101.stdout:add osd0 94370ba8-ec13-47e6-a0fe-d89f22ea7b4d 2023-01-25T09:54:19.351 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:54:19.392 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.387+0000 7fa794568580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:19.394 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.389+0000 7fa794568580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:19.395 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.390+0000 7fa794568580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:19.395 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.391+0000 7fa794568580 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-01-25T09:54:19.396 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.391+0000 7fa794568580 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-01-25T09:54:19.396 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.391+0000 7fa794568580 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-01-25T09:54:19.460 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:19.456+0000 7fa794568580 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2023-01-25T09:54:24.774 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd0 key to auth repository 2023-01-25T09:54:24.969 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.0 2023-01-25T09:54:24.996 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:24.992+0000 7f0806d3c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:25.000 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:24.995+0000 7f0806d3c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:25.001 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:24.996+0000 7f0806d3c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:25.181 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:54:25.525 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:25.521+0000 7f0806d3c580 -1 Falling back to public interface 2023-01-25T09:54:26.393 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:54:27.697 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:54:27.834 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:27.830+0000 7f0806d3c580 -1 osd.0 0 log_to_monitors true 2023-01-25T09:54:29.043 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:54:30.357 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:54:31.670 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:54:32.982 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:54:34.295 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:54:35.657 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:54:36.968 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:54:38.284 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:54:39.635 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:54:41.356 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:54:42.951 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:54:44.344 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:54:45.780 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:54:47.551 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:54:49.622 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:54:51.083 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T09:54:51.439 INFO:tasks.workunit.client.0.smithi101.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/100535,v1:127.0.0.1:6801/100535] [v2:127.0.0.1:6802/100535,v1:127.0.0.1:6803/100535] exists,up 94370ba8-ec13-47e6-a0fe-d89f22ea7b4d 2023-01-25T09:54:51.442 INFO:tasks.workunit.client.0.smithi101.stdout:add osd1 7ac449c6-ecaf-4a34-8f9f-bbeb14153530 2023-01-25T09:54:51.836 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:54:51.877 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:51.871+0000 7fe491840580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:51.900 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:51.889+0000 7fe491840580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:51.901 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:51.896+0000 7fe491840580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:51.902 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:51.896+0000 7fe491840580 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-01-25T09:54:51.902 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:51.896+0000 7fe491840580 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-01-25T09:54:51.902 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:51.896+0000 7fe491840580 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-01-25T09:54:52.055 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:52.050+0000 7fe491840580 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2023-01-25T09:54:57.582 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd1 key to auth repository 2023-01-25T09:54:57.975 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.1 2023-01-25T09:54:58.001 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:57.996+0000 7f9a8cf03580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:58.003 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:57.998+0000 7f9a8cf03580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:58.004 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:57.999+0000 7f9a8cf03580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:54:58.311 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:54:58.518 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:54:58.513+0000 7f9a8cf03580 -1 Falling back to public interface 2023-01-25T09:54:59.615 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:55:00.649 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:00.643+0000 7f9a8cf03580 -1 osd.1 0 log_to_monitors true 2023-01-25T09:55:00.935 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:55:02.297 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:55:03.644 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:55:04.959 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:55:06.268 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:55:07.724 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:55:09.086 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:55:10.470 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:55:11.777 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:55:13.256 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:55:14.842 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:55:16.245 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:55:17.847 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:55:19.181 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:55:20.686 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:55:22.253 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:55:22.567 INFO:tasks.workunit.client.0.smithi101.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/102435,v1:127.0.0.1:6811/102435] [v2:127.0.0.1:6812/102435,v1:127.0.0.1:6813/102435] exists,up 7ac449c6-ecaf-4a34-8f9f-bbeb14153530 2023-01-25T09:55:22.571 INFO:tasks.workunit.client.0.smithi101.stdout:add osd2 90e6cff3-e6aa-4c7e-a9da-b19153268fef 2023-01-25T09:55:22.999 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:55:23.040 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.036+0000 7f7d48e07580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:55:23.049 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.044+0000 7f7d48e07580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:55:23.050 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.045+0000 7f7d48e07580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:55:23.050 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.046+0000 7f7d48e07580 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-01-25T09:55:23.051 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.046+0000 7f7d48e07580 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-01-25T09:55:23.051 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.046+0000 7f7d48e07580 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-01-25T09:55:23.120 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:23.116+0000 7f7d48e07580 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2023-01-25T09:55:29.303 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd2 key to auth repository 2023-01-25T09:55:29.695 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.2 2023-01-25T09:55:29.722 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:29.717+0000 7fa5bb1ca580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:55:29.724 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:29.720+0000 7fa5bb1ca580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:55:29.725 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:29.721+0000 7fa5bb1ca580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:55:30.057 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:55:30.248 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:30.243+0000 7fa5bb1ca580 -1 Falling back to public interface 2023-01-25T09:55:31.389 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:55:32.654 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:55:32.648+0000 7fa5bb1ca580 -1 osd.2 0 log_to_monitors true 2023-01-25T09:55:32.703 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:55:34.045 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:55:35.402 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:55:36.716 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:55:38.077 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:55:39.432 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:55:40.742 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:55:42.058 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:55:43.399 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:55:44.711 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:55:46.028 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:55:47.435 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:55:49.285 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:55:50.909 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:55:52.386 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:55:53.735 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:55:55.098 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T09:55:55.402 INFO:tasks.workunit.client.0.smithi101.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/104182,v1:127.0.0.1:6819/104182] [v2:127.0.0.1:6820/104182,v1:127.0.0.1:6821/104182] exists,up 90e6cff3-e6aa-4c7e-a9da-b19153268fef 2023-01-25T09:55:55.836 INFO:tasks.workunit.client.0.smithi101.stderr:pool 'test' created 2023-01-25T09:55:57.643 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.0 seq 21474836500 2023-01-25T09:55:57.947 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.1 seq 42949672973 2023-01-25T09:55:58.273 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.2 seq 64424509448 2023-01-25T09:56:00.970 INFO:tasks.workunit.client.0.smithi101.stdout:standard_scrub_cluster: dbg: test pool is test 1 2023-01-25T09:56:00.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2023-01-25T09:56:01.124 INFO:tasks.workunit.client.0.smithi101.stdout:osd.0: { 2023-01-25T09:56:01.125 INFO:tasks.workunit.client.0.smithi101.stdout: "success": "osd_mclock_scheduler_background_best_effort_res = '13' osd_mclock_scheduler_background_recovery_lim = '52' osd_mclock_scheduler_background_recovery_res = '13' osd_mclock_scheduler_client_res = '26' osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T09:56:01.125 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:01.139 INFO:tasks.workunit.client.0.smithi101.stdout:osd.1: { 2023-01-25T09:56:01.139 INFO:tasks.workunit.client.0.smithi101.stdout: "success": "osd_mclock_scheduler_background_best_effort_res = '14' osd_mclock_scheduler_background_recovery_lim = '58' osd_mclock_scheduler_background_recovery_res = '14' osd_mclock_scheduler_client_res = '29' osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T09:56:01.140 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:01.154 INFO:tasks.workunit.client.0.smithi101.stdout:osd.2: { 2023-01-25T09:56:01.154 INFO:tasks.workunit.client.0.smithi101.stdout: "success": "osd_mclock_scheduler_background_best_effort_res = '14' osd_mclock_scheduler_background_recovery_lim = '56' osd_mclock_scheduler_background_recovery_res = '14' osd_mclock_scheduler_client_res = '28' osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T09:56:01.154 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:01.164 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2023-01-25T09:56:01.284 INFO:tasks.workunit.client.0.smithi101.stdout:osd.0: { 2023-01-25T09:56:01.284 INFO:tasks.workunit.client.0.smithi101.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T09:56:01.284 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:01.298 INFO:tasks.workunit.client.0.smithi101.stdout:osd.1: { 2023-01-25T09:56:01.298 INFO:tasks.workunit.client.0.smithi101.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T09:56:01.298 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:01.312 INFO:tasks.workunit.client.0.smithi101.stdout:osd.2: { 2023-01-25T09:56:01.313 INFO:tasks.workunit.client.0.smithi101.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T09:56:01.313 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:01.324 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2023-01-25T09:56:01.325 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2023-01-25T09:56:01.325 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2023-01-25T09:56:01.325 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2023-01-25T09:56:01.325 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2023-01-25T09:56:01.325 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-01-25T09:56:01.326 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-01-25T09:56:01.550 INFO:tasks.workunit.client.0.smithi101.stdout:created pool test snap snap01 2023-01-25T09:56:01.850 INFO:tasks.workunit.client.0.smithi101.stdout:created pool test snap snap02 2023-01-25T09:56:01.852 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2023-01-25T09:56:01.853 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-01-25T09:56:01.853 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-01-25T09:56:02.312 INFO:tasks.workunit.client.0.smithi101.stdout:created pool test snap snap13 2023-01-25T09:56:02.314 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2023-01-25T09:56:02.315 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-01-25T09:56:02.315 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-01-25T09:56:02.665 INFO:tasks.workunit.client.0.smithi101.stdout:created pool test snap snap24 2023-01-25T09:56:02.932 INFO:tasks.workunit.client.0.smithi101.stdout:created pool test snap snap25 2023-01-25T09:56:02.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 140 2023-01-25T09:56:02.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2023-01-25T09:56:03.244 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-01-25T09:56:03.245 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2023-01-25T09:56:03.548 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2023-01-25T09:56:03.548 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-01-25T09:56:03.549 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2023-01-25T09:56:03.877 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2023-01-25T09:56:03.878 INFO:tasks.workunit.client.0.smithi101.stdout:pgid is 1.3 (primary: osd.1) 2023-01-25T09:56:03.878 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2023-01-25T09:56:03.878 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2023-01-25T09:56:03.879 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:293: set_query_debug: local pgid=1.3 2023-01-25T09:56:03.879 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: ceph pg dump pgs_brief 2023-01-25T09:56:03.879 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2023-01-25T09:56:04.171 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs_brief 2023-01-25T09:56:04.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local prim_osd=1 2023-01-25T09:56:04.182 INFO:tasks.workunit.client.0.smithi101.stdout:Setting scrub debug data. Primary for 1.3 is 1 2023-01-25T09:56:04.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2023-01-25T09:56:04.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: get_asok_path osd.1 2023-01-25T09:56:04.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-25T09:56:04.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-25T09:56:04.184 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T09:56:04.185 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:56:04.185 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:56:04.185 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.99476/ceph-osd.1.asok 2023-01-25T09:56:04.185 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: CEPH_ARGS= 2023-01-25T09:56:04.185 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.99476/ceph-osd.1.asok scrubdebug 1.3 set sessions 2023-01-25T09:56:04.295 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:04.296 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-01-25T09:56:04.323 INFO:tasks.workunit.client.0.smithi101.stdout:{"success":true}{ 2023-01-25T09:56:04.324 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "objxxx", 2023-01-25T09:56:04.324 INFO:tasks.workunit.client.0.smithi101.stdout: "clones": [ 2023-01-25T09:56:04.325 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.325 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 2, 2023-01-25T09:56:04.326 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [ 2023-01-25T09:56:04.326 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.326 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 1, 2023-01-25T09:56:04.326 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap01" 2023-01-25T09:56:04.327 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:04.327 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.327 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 2, 2023-01-25T09:56:04.327 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap02" 2023-01-25T09:56:04.327 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:04.328 INFO:tasks.workunit.client.0.smithi101.stdout: ], 2023-01-25T09:56:04.328 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 4, 2023-01-25T09:56:04.328 INFO:tasks.workunit.client.0.smithi101.stdout: "overlaps": [] 2023-01-25T09:56:04.328 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:04.329 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.329 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 3, 2023-01-25T09:56:04.329 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [ 2023-01-25T09:56:04.330 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.330 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 3, 2023-01-25T09:56:04.331 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap13" 2023-01-25T09:56:04.331 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:04.331 INFO:tasks.workunit.client.0.smithi101.stdout: ], 2023-01-25T09:56:04.332 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 5, 2023-01-25T09:56:04.332 INFO:tasks.workunit.client.0.smithi101.stdout: "overlaps": [] 2023-01-25T09:56:04.333 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:04.333 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.333 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 5, 2023-01-25T09:56:04.333 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [ 2023-01-25T09:56:04.334 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.334 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 4, 2023-01-25T09:56:04.334 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap24" 2023-01-25T09:56:04.334 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:04.334 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.334 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 5, 2023-01-25T09:56:04.335 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap25" 2023-01-25T09:56:04.335 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:04.335 INFO:tasks.workunit.client.0.smithi101.stdout: ], 2023-01-25T09:56:04.335 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 6, 2023-01-25T09:56:04.335 INFO:tasks.workunit.client.0.smithi101.stdout: "overlaps": [] 2023-01-25T09:56:04.336 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:04.336 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:04.336 INFO:tasks.workunit.client.0.smithi101.stdout: "id": "head", 2023-01-25T09:56:04.336 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [], 2023-01-25T09:56:04.337 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 4 2023-01-25T09:56:04.337 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:04.337 INFO:tasks.workunit.client.0.smithi101.stdout: ] 2023-01-25T09:56:04.337 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:04.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-01-25T09:56:04.439 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:04.439 INFO:tasks.workunit.client.0.smithi101.stdout: "deep": true, 2023-01-25T09:56:04.440 INFO:tasks.workunit.client.0.smithi101.stdout: "stamp": "2023-01-18T09:54:24.435646+0000" 2023-01-25T09:56:04.440 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:04.449 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2023-01-25T09:56:07.451 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T09:56:07.745 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T09:56:07.746 INFO:tasks.workunit.client.0.smithi101.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2023-01-25T09:55:56.838481+0000 25'7 25:20 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:07.746 INFO:tasks.workunit.client.0.smithi101.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:55:56.838484+0000 0'0 20:12 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:07.746 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:55:56.838416+0000 0'0 25:17 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:07.746 INFO:tasks.workunit.client.0.smithi101.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:55:56.805786+0000 0'0 25:16 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:07.746 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:07.747 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T09:56:07.748 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T09:56:07.757 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2023-01-25T09:56:07.759 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T09:56:08.043 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T09:56:08.043 INFO:tasks.workunit.client.0.smithi101.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2023-01-25T09:55:56.838481+0000 25'7 25:20 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:08.043 INFO:tasks.workunit.client.0.smithi101.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:55:56.838484+0000 0'0 25:17 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:08.044 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:55:56.838416+0000 0'0 25:17 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:08.044 INFO:tasks.workunit.client.0.smithi101.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:55:56.805786+0000 0'0 25:16 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:08.044 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:08.044 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T09:56:08.045 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T09:56:08.055 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2023-01-25T09:56:08.521 INFO:tasks.workunit.client.0.smithi101.stderr:noscrub is set 2023-01-25T09:56:08.530 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2023-01-25T09:56:09.056 INFO:tasks.workunit.client.0.smithi101.stderr:nodeep-scrub is set 2023-01-25T09:56:09.065 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2023-01-25T09:56:14.067 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2023-01-25T09:56:14.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-01-25T09:56:14.068 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T09:56:14.069 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T09:56:14.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T09:56:14.070 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T09:56:14.070 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T09:56:14.378 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T09:56:14.378 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:14.378 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-01-25T09:56:14.404 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bdev(0x559e3cc72000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T09:56:14.405 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bdev(0x559e3cc72000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:14.405 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T09:56:14.405 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bdev(0x559e3cc72380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T09:56:14.405 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bdev(0x559e3cc72380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:14.405 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T09:56:14.406 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.399+0000 7fefb9cef8c0 1 bluefs mount 2023-01-25T09:56:14.412 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.407+0000 7fefb9cef8c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:14.443 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.438+0000 7fefb9cef8c0 1 bluefs mount shared_bdev_used = 0 2023-01-25T09:56:14.443 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.438+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:14.445 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.439+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:14.445 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:14.445 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:14.445 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:14.446 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:14.446 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:14.446 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:14.446 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:14.446 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:14.447 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:14.447 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:14.447 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.440+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:14.470 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.465+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:14.471 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.465+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2023-01-25T09:56:14.471 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.465+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-01-25T09:56:14.471 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.465+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T09:56:14.471 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.465+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-01-25T09:56:14.471 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.466+0000 7fefb9cef8c0 1 freelist init 2023-01-25T09:56:14.472 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.466+0000 7fefb9cef8c0 1 freelist _read_cfg 2023-01-25T09:56:14.472 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.466+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 1.14441e-07 2023-01-25T09:56:14.472 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.467+0000 7fefb9cef8c0 1 bluefs umount 2023-01-25T09:56:14.473 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.467+0000 7fefb9cef8c0 1 bdev(0x559e3cc72380 td/osd-mapper/0/block) close 2023-01-25T09:56:14.669 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.665+0000 7fefb9cef8c0 1 bdev(0x559e3cc72380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T09:56:14.670 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.665+0000 7fefb9cef8c0 1 bdev(0x559e3cc72380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:14.670 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.665+0000 7fefb9cef8c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T09:56:14.670 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.665+0000 7fefb9cef8c0 1 bluefs mount 2023-01-25T09:56:14.670 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.665+0000 7fefb9cef8c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:14.672 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.666+0000 7fefb9cef8c0 1 bluefs mount shared_bdev_used = 20512768 2023-01-25T09:56:14.672 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.666+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:14.672 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:14.673 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:14.673 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:14.673 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:14.673 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:14.673 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:14.674 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:14.674 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:14.674 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:14.674 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:14.674 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:14.675 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:14.667+0000 7fefb9cef8c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:15.344 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.339+0000 7fefb9cef8c0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:15.346 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:15.346 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:15.346 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:15.347 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:15.347 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T09:56:15.347 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:15.347 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:15.348 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.348 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:15.348 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T09:56:15.348 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:15.348 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:15.349 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T09:56:15.349 INFO:tasks.workunit.client.0.smithi101.stdout:0000003f 2023-01-25T09:56:15.349 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.349 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:15.349 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:15.350 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:15.350 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:15.350 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T09:56:15.350 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:15.350 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:15.351 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.351 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:15.351 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:15.351 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:15.351 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:15.351 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:15.352 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:15.352 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.352 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:15.352 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:15.352 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:15.353 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:15.353 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:15.353 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:15.353 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.353 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:15.354 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:15.354 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:15.354 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:15.354 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:15.354 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:15.355 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.355 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:15.355 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:15.355 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:15.355 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:15.355 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:15.356 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:15.356 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.356 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:15.356 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:15.356 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:15.357 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:15.357 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:15.357 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:15.357 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.357 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T09:56:15.358 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.358 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T09:56:15.358 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:15.358 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.339+0000 7fefb9cef8c0 1 bluefs umount 2023-01-25T09:56:15.359 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.339+0000 7fefb9cef8c0 1 bdev(0x559e3cc72380 td/osd-mapper/0/block) close 2023-01-25T09:56:15.580 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.576+0000 7fefb9cef8c0 1 freelist shutdown 2023-01-25T09:56:15.581 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.576+0000 7fefb9cef8c0 1 bdev(0x559e3cc72000 td/osd-mapper/0/block) close 2023-01-25T09:56:15.670 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:15.670 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T09:56:15.671 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2023-01-25T09:56:15.697 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T09:56:15.698 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:15.698 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T09:56:15.698 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T09:56:15.698 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:15.699 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T09:56:15.699 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.692+0000 7eff9fdd28c0 1 bluefs mount 2023-01-25T09:56:15.711 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.705+0000 7eff9fdd28c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:15.717 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.712+0000 7eff9fdd28c0 1 bluefs mount shared_bdev_used = 0 2023-01-25T09:56:15.718 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.712+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:15.720 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:15.720 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:15.721 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:15.721 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:15.721 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:15.721 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:15.721 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.713+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:15.722 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.714+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:15.722 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.714+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:15.722 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.714+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:15.722 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.714+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:15.722 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.714+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:15.743 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:15.744 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-01-25T09:56:15.744 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-01-25T09:56:15.744 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T09:56:15.744 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-01-25T09:56:15.745 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 freelist init 2023-01-25T09:56:15.745 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.738+0000 7eff9fdd28c0 1 freelist _read_cfg 2023-01-25T09:56:15.745 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.739+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 1.14441e-07 2023-01-25T09:56:15.746 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.739+0000 7eff9fdd28c0 1 bluefs umount 2023-01-25T09:56:15.746 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.739+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8380 td/osd-mapper/2/block) close 2023-01-25T09:56:15.963 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.957+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T09:56:15.963 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.957+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:15.963 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.957+0000 7eff9fdd28c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T09:56:15.963 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.957+0000 7eff9fdd28c0 1 bluefs mount 2023-01-25T09:56:15.977 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.972+0000 7eff9fdd28c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:15.978 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.972+0000 7eff9fdd28c0 1 bluefs mount shared_bdev_used = 20447232 2023-01-25T09:56:15.978 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.973+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:15.980 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.973+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:15.980 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.973+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:15.980 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.973+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:15.980 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.973+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:15.981 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:15.981 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:15.981 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:15.981 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:15.981 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:15.982 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:15.982 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:15.982 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:15.974+0000 7eff9fdd28c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:16.755 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:16.749+0000 7eff9fdd28c0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:16.755 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:16.750+0000 7eff9fdd28c0 1 bluefs umount 2023-01-25T09:56:16.756 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:16.750+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8380 td/osd-mapper/2/block) close 2023-01-25T09:56:16.991 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:16.986+0000 7eff9fdd28c0 1 freelist shutdown 2023-01-25T09:56:16.991 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:16.986+0000 7eff9fdd28c0 1 bdev(0x55c8f56e8000 td/osd-mapper/2/block) close 2023-01-25T09:56:17.217 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:17.217 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:17.218 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:17.218 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:17.218 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:17.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:17.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2023-01-25T09:56:17.219 INFO:tasks.workunit.client.0.smithi101.stderr:grep: /tmp/oo2.dump: No such file or directory 2023-01-25T09:56:17.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:17.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T09:56:17.816 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:17.816 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:17.816 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:17.817 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:17.817 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T09:56:17.817 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:17.817 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:17.818 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.818 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:17.818 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T09:56:17.818 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:17.818 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:17.819 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T09:56:17.819 INFO:tasks.workunit.client.0.smithi101.stdout:0000003f 2023-01-25T09:56:17.819 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.819 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:17.819 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:17.820 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:17.820 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:17.820 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T09:56:17.820 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:17.820 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:17.821 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.821 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:17.821 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:17.821 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:17.821 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:17.822 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:17.822 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:17.822 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.822 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:17.822 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:17.823 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:17.823 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:17.823 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:17.823 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:17.823 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.823 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:17.824 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:17.824 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:17.824 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:17.824 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:17.824 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:17.825 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.825 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:17.825 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:17.825 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:17.825 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:17.826 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:17.826 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:17.826 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.826 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:17.826 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:17.827 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:17.827 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:17.827 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:17.827 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:17.827 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.828 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T09:56:17.828 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:17.828 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T09:56:17.828 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:18.260 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T09:56:18.260 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-01-25T09:56:18.260 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T09:56:18.261 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2023-01-25T09:56:19.551 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2023-01-25T09:56:19.552 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2023-01-25T09:56:19.553 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2023-01-25T09:56:19.554 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:19.554 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2023-01-25T09:56:19.554 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2023-01-25T09:56:19.555 INFO:tasks.workunit.client.0.smithi101.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2023-01-25T09:56:19.555 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-01-25T09:56:19.556 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-01-25T09:56:19.580 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bdev(0x55e5d6abe000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T09:56:19.580 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bdev(0x55e5d6abe000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:19.580 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T09:56:19.581 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bdev(0x55e5d6abe380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T09:56:19.581 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bdev(0x55e5d6abe380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:19.581 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T09:56:19.581 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.574+0000 7f233e1b48c0 1 bluefs mount 2023-01-25T09:56:19.586 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.581+0000 7f233e1b48c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:19.593 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.587+0000 7f233e1b48c0 1 bluefs mount shared_bdev_used = 0 2023-01-25T09:56:19.593 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.587+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:19.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:19.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:19.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:19.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:19.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:19.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:19.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:19.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:19.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:19.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:19.597 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:19.597 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.589+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:19.599 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:19.599 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2023-01-25T09:56:19.600 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-01-25T09:56:19.600 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T09:56:19.600 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-01-25T09:56:19.600 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 freelist init 2023-01-25T09:56:19.600 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.594+0000 7f233e1b48c0 1 freelist _read_cfg 2023-01-25T09:56:19.601 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.595+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 1.14441e-07 2023-01-25T09:56:19.601 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.595+0000 7f233e1b48c0 1 bluefs umount 2023-01-25T09:56:19.602 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.595+0000 7f233e1b48c0 1 bdev(0x55e5d6abe380 td/osd-mapper/0/block) close 2023-01-25T09:56:19.841 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.836+0000 7f233e1b48c0 1 bdev(0x55e5d6abe380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T09:56:19.842 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.836+0000 7f233e1b48c0 1 bdev(0x55e5d6abe380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:19.842 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.836+0000 7f233e1b48c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T09:56:19.842 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.836+0000 7f233e1b48c0 1 bluefs mount 2023-01-25T09:56:19.847 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.841+0000 7f233e1b48c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:19.848 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.842+0000 7f233e1b48c0 1 bluefs mount shared_bdev_used = 6815744 2023-01-25T09:56:19.848 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.842+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:19.849 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.843+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:19.850 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.843+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:19.850 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.843+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:19.850 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.843+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:19.850 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.843+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:19.850 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.843+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:19.851 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.844+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:19.851 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.844+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:19.851 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.844+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:19.851 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.844+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:19.851 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.844+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:19.852 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:19.844+0000 7f233e1b48c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:20.329 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:20.324+0000 7f233e1b48c0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:20.330 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:20.331 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:20.331 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:20.331 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:20.331 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T09:56:20.332 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:20.332 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:20.332 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.332 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:20.332 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T09:56:20.333 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:20.333 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:20.333 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T09:56:20.333 INFO:tasks.workunit.client.0.smithi101.stdout:0000003f 2023-01-25T09:56:20.333 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.334 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:20.334 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:20.334 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:20.334 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:20.334 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T09:56:20.335 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:20.335 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:20.335 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.335 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:20.335 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:20.336 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:20.336 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:20.336 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:20.336 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:20.336 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.337 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:20.337 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:20.337 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:20.337 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:20.338 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:20.338 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:20.338 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.338 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:20.338 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:20.339 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:20.339 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:20.339 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:20.339 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:20.340 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.340 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:20.340 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:20.340 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:20.341 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:20.341 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:20.341 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:20.341 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.341 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:20.342 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:20.342 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:20.342 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:20.342 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:20.342 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:20.342 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.343 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T09:56:20.343 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.343 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T09:56:20.343 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:20.344 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:20.324+0000 7f233e1b48c0 1 bluefs umount 2023-01-25T09:56:20.344 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:20.324+0000 7f233e1b48c0 1 bdev(0x55e5d6abe380 td/osd-mapper/0/block) close 2023-01-25T09:56:20.565 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:20.560+0000 7f233e1b48c0 1 freelist shutdown 2023-01-25T09:56:20.565 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:20.560+0000 7f233e1b48c0 1 bdev(0x55e5d6abe000 td/osd-mapper/0/block) close 2023-01-25T09:56:20.595 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-01-25T09:56:20.596 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-01-25T09:56:20.596 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T09:56:21.643 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:21.644 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:21.644 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-01-25T09:56:21.644 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-01-25T09:56:21.645 INFO:tasks.workunit.client.0.smithi101.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:21.645 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-01-25T09:56:21.646 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.ucJFccsZnG_the_val 2023-01-25T09:56:21.646 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:21.646 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.ucJFccsZnG_the_val' 2023-01-25T09:56:21.647 INFO:tasks.workunit.client.0.smithi101.stdout:Value dumped in: /tmp/tmp.ucJFccsZnG_the_val 2023-01-25T09:56:21.647 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.ucJFccsZnG_the_val 2023-01-25T09:56:22.342 INFO:tasks.workunit.client.0.smithi101.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-01-25T09:56:22.690 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:22.690 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.ucJFccsZnG_the_val 2023-01-25T09:56:22.695 INFO:tasks.workunit.client.0.smithi101.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-01-25T09:56:22.695 INFO:tasks.workunit.client.0.smithi101.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-01-25T09:56:22.695 INFO:tasks.workunit.client.0.smithi101.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-01-25T09:56:22.696 INFO:tasks.workunit.client.0.smithi101.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-01-25T09:56:22.696 INFO:tasks.workunit.client.0.smithi101.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-01-25T09:56:22.696 INFO:tasks.workunit.client.0.smithi101.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-01-25T09:56:22.697 INFO:tasks.workunit.client.0.smithi101.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-01-25T09:56:22.697 INFO:tasks.workunit.client.0.smithi101.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-01-25T09:56:22.698 INFO:tasks.workunit.client.0.smithi101.stdout:0000073 2023-01-25T09:56:22.698 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T09:56:22.699 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:23.991 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.ucJFccsZnG_the_val 2023-01-25T09:56:25.037 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:25.037 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:25.037 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-01-25T09:56:25.038 INFO:tasks.workunit.client.0.smithi101.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2023-01-25T09:56:25.038 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2023-01-25T09:56:25.038 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-01-25T09:56:25.039 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T09:56:25.063 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.057+0000 7f73033c88c0 1 bdev(0x55fdb7c46000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-01-25T09:56:25.063 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.057+0000 7f73033c88c0 1 bdev(0x55fdb7c46000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:25.063 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.057+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T09:56:25.064 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.057+0000 7f73033c88c0 1 bdev(0x55fdb7c46380 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-01-25T09:56:25.064 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.058+0000 7f73033c88c0 1 bdev(0x55fdb7c46380 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:25.064 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.058+0000 7f73033c88c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-01-25T09:56:25.064 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.058+0000 7f73033c88c0 1 bluefs mount 2023-01-25T09:56:25.066 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.061+0000 7f73033c88c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:25.073 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.067+0000 7f73033c88c0 1 bluefs mount shared_bdev_used = 0 2023-01-25T09:56:25.073 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.068+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:25.075 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:25.075 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:25.075 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:25.075 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:25.076 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:25.076 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:25.076 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:25.076 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:25.076 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:25.077 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:25.077 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:25.077 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.069+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:25.079 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:25.079 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2023-01-25T09:56:25.080 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2023-01-25T09:56:25.080 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T09:56:25.080 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2023-01-25T09:56:25.081 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 freelist init 2023-01-25T09:56:25.081 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.074+0000 7f73033c88c0 1 freelist _read_cfg 2023-01-25T09:56:25.081 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.075+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 1.14441e-07 2023-01-25T09:56:25.082 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.075+0000 7f73033c88c0 1 bluefs umount 2023-01-25T09:56:25.082 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.075+0000 7f73033c88c0 1 bdev(0x55fdb7c46380 td/osd-mapper/1/block) close 2023-01-25T09:56:25.325 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.320+0000 7f73033c88c0 1 bdev(0x55fdb7c46380 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-01-25T09:56:25.326 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.320+0000 7f73033c88c0 1 bdev(0x55fdb7c46380 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:25.326 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.320+0000 7f73033c88c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-01-25T09:56:25.326 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.320+0000 7f73033c88c0 1 bluefs mount 2023-01-25T09:56:25.337 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.327+0000 7f73033c88c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:25.338 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.328+0000 7f73033c88c0 1 bluefs mount shared_bdev_used = 6750208 2023-01-25T09:56:25.338 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.328+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:25.339 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.329+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:25.339 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.329+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:25.339 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.329+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:25.339 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.329+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:25.339 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.329+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:25.340 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.329+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:25.340 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.330+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:25.340 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.330+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:25.341 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.330+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:25.341 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.330+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:25.341 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.330+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:25.342 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.330+0000 7f73033c88c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:25.713 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.708+0000 7f73033c88c0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:25.716 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:25.717 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:25.717 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:25.717 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:25.718 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T09:56:25.718 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:25.719 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:25.719 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.719 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:25.719 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T09:56:25.720 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:25.720 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:25.721 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T09:56:25.721 INFO:tasks.workunit.client.0.smithi101.stdout:0000003f 2023-01-25T09:56:25.721 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.722 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:25.722 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:25.722 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:25.722 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:25.723 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T09:56:25.723 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:25.723 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:25.723 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.723 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:25.724 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:25.724 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:25.724 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:25.724 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:25.725 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:25.725 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.725 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:25.725 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:25.726 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:25.726 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:25.726 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:25.726 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:25.727 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.727 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:25.727 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:25.727 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:25.727 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:25.728 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:25.728 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:25.728 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.728 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:25.728 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:25.729 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:25.729 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:25.729 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:25.729 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:25.729 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.730 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:25.730 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:25.730 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:25.730 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:25.730 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:25.731 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:25.731 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.731 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T09:56:25.731 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.731 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T09:56:25.732 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:25.732 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.709+0000 7f73033c88c0 1 bluefs umount 2023-01-25T09:56:25.732 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.709+0000 7f73033c88c0 1 bdev(0x55fdb7c46380 td/osd-mapper/1/block) close 2023-01-25T09:56:25.953 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.948+0000 7f73033c88c0 1 freelist shutdown 2023-01-25T09:56:25.954 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:25.948+0000 7f73033c88c0 1 bdev(0x55fdb7c46000 td/osd-mapper/1/block) close 2023-01-25T09:56:26.083 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T09:56:26.083 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-01-25T09:56:26.083 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T09:56:27.123 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:27.124 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:27.124 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-01-25T09:56:27.124 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-01-25T09:56:27.125 INFO:tasks.workunit.client.0.smithi101.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:27.125 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-01-25T09:56:27.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.aS40pFD5k7_the_val 2023-01-25T09:56:27.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:27.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.aS40pFD5k7_the_val' 2023-01-25T09:56:27.126 INFO:tasks.workunit.client.0.smithi101.stdout:Value dumped in: /tmp/tmp.aS40pFD5k7_the_val 2023-01-25T09:56:27.127 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.aS40pFD5k7_the_val 2023-01-25T09:56:27.792 INFO:tasks.workunit.client.0.smithi101.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-01-25T09:56:28.170 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:28.170 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.aS40pFD5k7_the_val 2023-01-25T09:56:28.171 INFO:tasks.workunit.client.0.smithi101.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-01-25T09:56:28.171 INFO:tasks.workunit.client.0.smithi101.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-01-25T09:56:28.171 INFO:tasks.workunit.client.0.smithi101.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-01-25T09:56:28.172 INFO:tasks.workunit.client.0.smithi101.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-01-25T09:56:28.172 INFO:tasks.workunit.client.0.smithi101.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-01-25T09:56:28.172 INFO:tasks.workunit.client.0.smithi101.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-01-25T09:56:28.172 INFO:tasks.workunit.client.0.smithi101.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-01-25T09:56:28.172 INFO:tasks.workunit.client.0.smithi101.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-01-25T09:56:28.173 INFO:tasks.workunit.client.0.smithi101.stdout:0000073 2023-01-25T09:56:28.173 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T09:56:28.173 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:29.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.aS40pFD5k7_the_val 2023-01-25T09:56:30.258 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:30.258 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:30.258 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2023-01-25T09:56:30.259 INFO:tasks.workunit.client.0.smithi101.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2023-01-25T09:56:30.259 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2023-01-25T09:56:30.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-01-25T09:56:30.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T09:56:30.283 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.277+0000 7f97cc3368c0 1 bdev(0x5597aec5c000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T09:56:30.284 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.278+0000 7f97cc3368c0 1 bdev(0x5597aec5c000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:30.284 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.278+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T09:56:30.284 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.278+0000 7f97cc3368c0 1 bdev(0x5597aec5c380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T09:56:30.284 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.278+0000 7f97cc3368c0 1 bdev(0x5597aec5c380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:30.285 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.278+0000 7f97cc3368c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T09:56:30.285 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.278+0000 7f97cc3368c0 1 bluefs mount 2023-01-25T09:56:30.292 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.287+0000 7f97cc3368c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:30.299 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 1 bluefs mount shared_bdev_used = 0 2023-01-25T09:56:30.300 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:30.301 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:30.302 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:30.302 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:30.302 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:30.302 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:30.302 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:30.303 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:30.303 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:30.303 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.295+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:30.303 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.296+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:30.303 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.296+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:30.304 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.296+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:30.307 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:30.307 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-01-25T09:56:30.308 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-01-25T09:56:30.308 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T09:56:30.308 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-01-25T09:56:30.308 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 freelist init 2023-01-25T09:56:30.309 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.302+0000 7f97cc3368c0 1 freelist _read_cfg 2023-01-25T09:56:30.309 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.303+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 1.14441e-07 2023-01-25T09:56:30.309 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.303+0000 7f97cc3368c0 1 bluefs umount 2023-01-25T09:56:30.310 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.303+0000 7f97cc3368c0 1 bdev(0x5597aec5c380 td/osd-mapper/2/block) close 2023-01-25T09:56:30.545 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.540+0000 7f97cc3368c0 1 bdev(0x5597aec5c380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T09:56:30.546 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.540+0000 7f97cc3368c0 1 bdev(0x5597aec5c380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T09:56:30.546 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.540+0000 7f97cc3368c0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T09:56:30.546 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.540+0000 7f97cc3368c0 1 bluefs mount 2023-01-25T09:56:30.547 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.540+0000 7f97cc3368c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T09:56:30.547 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.541+0000 7f97cc3368c0 1 bluefs mount shared_bdev_used = 6815744 2023-01-25T09:56:30.547 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.541+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T09:56:30.548 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.542+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T09:56:30.549 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.542+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T09:56:30.549 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.542+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T09:56:30.549 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.542+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T09:56:30.549 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.542+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T09:56:30.550 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.542+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T09:56:30.550 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.543+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T09:56:30.550 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.543+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T09:56:30.550 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.543+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T09:56:30.550 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.543+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T09:56:30.550 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.543+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T09:56:30.551 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.543+0000 7f97cc3368c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T09:56:30.865 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.861+0000 7f97cc3368c0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T09:56:30.875 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:30.875 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:30.875 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:30.875 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:30.876 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T09:56:30.876 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:30.876 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:30.876 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.876 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:30.877 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T09:56:30.877 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:30.877 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:30.877 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T09:56:30.877 INFO:tasks.workunit.client.0.smithi101.stdout:0000003f 2023-01-25T09:56:30.878 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.878 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:30.878 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T09:56:30.878 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T09:56:30.878 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T09:56:30.879 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T09:56:30.879 INFO:tasks.workunit.client.0.smithi101.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T09:56:30.879 INFO:tasks.workunit.client.0.smithi101.stdout:00000047 2023-01-25T09:56:30.879 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.879 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:30.880 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:30.880 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:30.880 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:30.880 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:30.880 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:30.881 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.881 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:56:30.881 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:30.881 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:30.881 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:30.882 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:30.882 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:30.882 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.882 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:30.883 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:30.883 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:30.883 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:30.883 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:30.883 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:30.884 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.884 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:30.884 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:30.884 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:30.884 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:30.884 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:30.885 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:30.885 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.885 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:56:30.885 INFO:tasks.workunit.client.0.smithi101.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T09:56:30.885 INFO:tasks.workunit.client.0.smithi101.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T09:56:30.886 INFO:tasks.workunit.client.0.smithi101.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T09:56:30.886 INFO:tasks.workunit.client.0.smithi101.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T09:56:30.886 INFO:tasks.workunit.client.0.smithi101.stdout:0000003b 2023-01-25T09:56:30.886 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.886 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T09:56:30.887 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.887 INFO:tasks.workunit.client.0.smithi101.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T09:56:30.887 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:30.887 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.861+0000 7f97cc3368c0 1 bluefs umount 2023-01-25T09:56:30.888 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:30.861+0000 7f97cc3368c0 1 bdev(0x5597aec5c380 td/osd-mapper/2/block) close 2023-01-25T09:56:31.102 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:31.097+0000 7f97cc3368c0 1 freelist shutdown 2023-01-25T09:56:31.102 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:31.097+0000 7f97cc3368c0 1 bdev(0x5597aec5c000 td/osd-mapper/2/block) close 2023-01-25T09:56:31.299 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T09:56:31.299 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-01-25T09:56:31.300 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T09:56:32.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:32.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:32.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-01-25T09:56:32.341 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-01-25T09:56:32.341 INFO:tasks.workunit.client.0.smithi101.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:32.342 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-01-25T09:56:32.342 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.gSufC376Y9_the_val 2023-01-25T09:56:32.342 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:32.343 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.gSufC376Y9_the_val' 2023-01-25T09:56:32.343 INFO:tasks.workunit.client.0.smithi101.stdout:Value dumped in: /tmp/tmp.gSufC376Y9_the_val 2023-01-25T09:56:32.343 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.gSufC376Y9_the_val 2023-01-25T09:56:33.003 INFO:tasks.workunit.client.0.smithi101.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-01-25T09:56:33.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:33.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.gSufC376Y9_the_val 2023-01-25T09:56:33.393 INFO:tasks.workunit.client.0.smithi101.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-01-25T09:56:33.393 INFO:tasks.workunit.client.0.smithi101.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-01-25T09:56:33.393 INFO:tasks.workunit.client.0.smithi101.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-01-25T09:56:33.394 INFO:tasks.workunit.client.0.smithi101.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-01-25T09:56:33.394 INFO:tasks.workunit.client.0.smithi101.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-01-25T09:56:33.394 INFO:tasks.workunit.client.0.smithi101.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-01-25T09:56:33.394 INFO:tasks.workunit.client.0.smithi101.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-01-25T09:56:33.394 INFO:tasks.workunit.client.0.smithi101.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-01-25T09:56:33.395 INFO:tasks.workunit.client.0.smithi101.stdout:0000073 2023-01-25T09:56:33.395 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T09:56:33.395 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:56:34.686 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.gSufC376Y9_the_val 2023-01-25T09:56:35.729 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:35.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off ' 2023-01-25T09:56:35.730 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off 2023-01-25T09:56:35.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:35.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:56:35.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/' 2023-01-25T09:56:35.731 INFO:tasks.workunit.client.0.smithi101.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/ 2023-01-25T09:56:35.732 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2023-01-25T09:56:35.732 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2023-01-25T09:56:35.736 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:35.737 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:35.737 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2023-01-25T09:56:35.737 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-mapper 2023-01-25T09:56:35.737 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2023-01-25T09:56:35.737 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2023-01-25T09:56:35.738 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2023-01-25T09:56:35.738 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-mapper/0 2023-01-25T09:56:35.738 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:35.738 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:56:35.739 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:56:35.739 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:56:35.739 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2023-01-25T09:56:35.739 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2023-01-25T09:56:35.740 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2023-01-25T09:56:35.740 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2023-01-25T09:56:35.740 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-01-25T09:56:35.740 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2023-01-25T09:56:35.740 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:56:35.741 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:56:35.741 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:56:35.741 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:56:35.742 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:56:35.742 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99476/$cluster-$name.asok' 2023-01-25T09:56:35.742 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99476/$cluster-$name.asok' 2023-01-25T09:56:35.742 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:56:35.742 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-01-25T09:56:35.743 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-01-25T09:56:35.743 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:56:35.743 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:56:35.743 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:56:35.743 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2023-01-25T09:56:35.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2023-01-25T09:56:35.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-mapper/0 2023-01-25T09:56:35.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2023-01-25T09:56:35.745 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.0 2023-01-25T09:56:35.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.99476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:56:35.746 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-mapper/0/whoami 2023-01-25T09:56:35.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2023-01-25T09:56:35.747 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2023-01-25T09:56:35.747 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2023-01-25T09:56:35.747 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2023-01-25T09:56:35.770 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:35.766+0000 7f3fd0612580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:35.772 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:35.768+0000 7f3fd0612580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:35.774 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:35.769+0000 7f3fd0612580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:36.058 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2023-01-25T09:56:36.059 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:56:36.059 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T09:56:36.059 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:56:36.060 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:56:36.060 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:36.061 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:56:36.061 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:56:36.061 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:36.062 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:56:36.292 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:36.288+0000 7f3fd0612580 -1 Falling back to public interface 2023-01-25T09:56:36.385 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:37.387 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:37.387 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:37.388 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:56:37.388 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:56:37.388 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:37.389 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:56:37.701 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:38.675 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:38.671+0000 7f3fd0612580 -1 osd.0 27 log_to_monitors true 2023-01-25T09:56:38.702 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:38.703 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:38.703 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:56:38.703 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:56:38.704 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:38.705 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:56:39.085 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:40.086 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:40.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:40.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T09:56:40.088 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:56:40.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:40.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:56:40.398 INFO:tasks.workunit.client.0.smithi101.stdout:osd.0 up in weight 1 up_from 30 up_thru 30 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6800/113107,v1:127.0.0.1:6801/113107] [v2:127.0.0.1:6802/113107,v1:127.0.0.1:6803/113107] exists,up 94370ba8-ec13-47e6-a0fe-d89f22ea7b4d 2023-01-25T09:56:40.402 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:56:40.402 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:56:40.403 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:56:40.403 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:40.403 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:40.403 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2023-01-25T09:56:40.403 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-mapper 2023-01-25T09:56:40.404 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2023-01-25T09:56:40.404 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2023-01-25T09:56:40.404 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2023-01-25T09:56:40.404 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-mapper/1 2023-01-25T09:56:40.404 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:40.405 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:56:40.405 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:56:40.405 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:56:40.405 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2023-01-25T09:56:40.405 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2023-01-25T09:56:40.406 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2023-01-25T09:56:40.406 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2023-01-25T09:56:40.406 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-01-25T09:56:40.406 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2023-01-25T09:56:40.406 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:56:40.406 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:56:40.407 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:56:40.407 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:56:40.407 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:56:40.407 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99476/$cluster-$name.asok' 2023-01-25T09:56:40.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99476/$cluster-$name.asok' 2023-01-25T09:56:40.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:56:40.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-01-25T09:56:40.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-01-25T09:56:40.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:56:40.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:56:40.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:56:40.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2023-01-25T09:56:40.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2023-01-25T09:56:40.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-mapper/1 2023-01-25T09:56:40.410 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2023-01-25T09:56:40.410 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.1 2023-01-25T09:56:40.411 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.99476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:56:40.411 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-mapper/1/whoami 2023-01-25T09:56:40.411 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2023-01-25T09:56:40.412 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2023-01-25T09:56:40.412 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2023-01-25T09:56:40.412 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2023-01-25T09:56:40.433 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:40.428+0000 7fece8922580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:40.434 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:40.430+0000 7fece8922580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:40.435 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:40.431+0000 7fece8922580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:40.757 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2023-01-25T09:56:40.758 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:56:40.758 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T09:56:40.758 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:56:40.758 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:56:40.759 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:40.759 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:56:40.759 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:56:40.760 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:40.760 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:56:40.956 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:40.952+0000 7fece8922580 -1 Falling back to public interface 2023-01-25T09:56:41.102 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:42.103 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:42.104 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:42.104 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:56:42.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:56:42.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:42.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:56:42.426 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:43.347 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:43.343+0000 7fece8922580 -1 osd.1 27 log_to_monitors true 2023-01-25T09:56:43.427 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:43.428 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:43.428 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:56:43.428 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:56:43.429 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:43.429 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:56:43.769 INFO:tasks.workunit.client.0.smithi101.stdout:osd.1 up in weight 1 up_from 33 up_thru 17 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/113676,v1:127.0.0.1:6811/113676] [v2:127.0.0.1:6812/113676,v1:127.0.0.1:6813/113676] exists,up 7ac449c6-ecaf-4a34-8f9f-bbeb14153530 2023-01-25T09:56:43.773 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:56:43.773 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:56:43.773 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:56:43.774 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:43.774 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:43.774 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2023-01-25T09:56:43.774 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-mapper 2023-01-25T09:56:43.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2023-01-25T09:56:43.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2023-01-25T09:56:43.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2023-01-25T09:56:43.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-mapper/2 2023-01-25T09:56:43.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T09:56:43.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:56:43.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:56:43.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:56:43.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2023-01-25T09:56:43.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2023-01-25T09:56:43.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2023-01-25T09:56:43.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2023-01-25T09:56:43.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-01-25T09:56:43.777 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2023-01-25T09:56:43.778 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:56:43.778 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:56:43.778 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:56:43.778 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:56:43.778 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:56:43.779 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99476/$cluster-$name.asok' 2023-01-25T09:56:43.779 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99476/$cluster-$name.asok' 2023-01-25T09:56:43.779 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:56:43.779 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-01-25T09:56:43.780 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-01-25T09:56:43.780 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:56:43.780 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:56:43.780 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:56:43.780 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2023-01-25T09:56:43.781 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2023-01-25T09:56:43.781 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-mapper/2 2023-01-25T09:56:43.781 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2023-01-25T09:56:43.781 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.2 2023-01-25T09:56:43.782 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=965596b4-40a6-4b8c-9f4a-9eb4b06177bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.99476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:56:43.783 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-mapper/2/whoami 2023-01-25T09:56:43.783 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2023-01-25T09:56:43.783 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2023-01-25T09:56:43.783 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2023-01-25T09:56:43.783 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2023-01-25T09:56:43.804 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:43.800+0000 7f2e63ed3580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:43.806 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:43.802+0000 7f2e63ed3580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:43.807 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:43.803+0000 7f2e63ed3580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:56:44.207 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2023-01-25T09:56:44.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:56:44.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T09:56:44.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:56:44.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:56:44.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:44.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:56:44.209 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:56:44.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:44.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:56:44.326 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:44.321+0000 7f2e63ed3580 -1 Falling back to public interface 2023-01-25T09:56:44.655 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:45.656 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:45.656 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:45.657 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:56:45.657 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:56:45.657 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:45.658 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:56:45.978 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:56:46.693 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:56:46.689+0000 7f2e63ed3580 -1 osd.2 27 log_to_monitors true 2023-01-25T09:56:46.979 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:56:46.980 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:56:46.980 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:56:46.981 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:56:46.981 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:56:46.982 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:56:47.381 INFO:tasks.workunit.client.0.smithi101.stdout:osd.2 up in weight 1 up_from 36 up_thru 17 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/114220,v1:127.0.0.1:6819/114220] [v2:127.0.0.1:6820/114220,v1:127.0.0.1:6821/114220] exists,up 90e6cff3-e6aa-4c7e-a9da-b19153268fef 2023-01-25T09:56:47.381 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:56:47.382 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:56:47.382 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:56:47.382 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2023-01-25T09:56:48.383 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2023-01-25T09:56:48.384 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2023-01-25T09:56:48.385 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:48.385 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2023-01-25T09:56:48.508 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:48.508 INFO:tasks.workunit.client.0.smithi101.stdout: "version": "18.0.0-1986-g44e4095d", 2023-01-25T09:56:48.508 INFO:tasks.workunit.client.0.smithi101.stdout: "release": "reef", 2023-01-25T09:56:48.509 INFO:tasks.workunit.client.0.smithi101.stdout: "release_type": "dev" 2023-01-25T09:56:48.509 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:48.518 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:48.518 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2023-01-25T09:56:48.637 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:48.637 INFO:tasks.workunit.client.0.smithi101.stdout: "version": "18.0.0-1986-g44e4095d", 2023-01-25T09:56:48.637 INFO:tasks.workunit.client.0.smithi101.stdout: "release": "reef", 2023-01-25T09:56:48.637 INFO:tasks.workunit.client.0.smithi101.stdout: "release_type": "dev" 2023-01-25T09:56:48.638 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:48.646 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T09:56:48.646 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2023-01-25T09:56:48.764 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:48.765 INFO:tasks.workunit.client.0.smithi101.stdout: "version": "18.0.0-1986-g44e4095d", 2023-01-25T09:56:48.765 INFO:tasks.workunit.client.0.smithi101.stdout: "release": "reef", 2023-01-25T09:56:48.766 INFO:tasks.workunit.client.0.smithi101.stdout: "release_type": "dev" 2023-01-25T09:56:48.766 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:48.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-01-25T09:56:48.804 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:48.804 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "objxxx", 2023-01-25T09:56:48.804 INFO:tasks.workunit.client.0.smithi101.stdout: "clones": [ 2023-01-25T09:56:48.805 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.805 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 2, 2023-01-25T09:56:48.805 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [ 2023-01-25T09:56:48.805 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.805 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 1, 2023-01-25T09:56:48.806 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap01" 2023-01-25T09:56:48.806 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:48.806 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.806 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 2, 2023-01-25T09:56:48.806 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap02" 2023-01-25T09:56:48.807 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:48.807 INFO:tasks.workunit.client.0.smithi101.stdout: ], 2023-01-25T09:56:48.807 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 4, 2023-01-25T09:56:48.807 INFO:tasks.workunit.client.0.smithi101.stdout: "overlaps": [] 2023-01-25T09:56:48.807 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:48.807 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.808 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 3, 2023-01-25T09:56:48.808 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [ 2023-01-25T09:56:48.808 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.808 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 3, 2023-01-25T09:56:48.808 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap13" 2023-01-25T09:56:48.809 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:48.809 INFO:tasks.workunit.client.0.smithi101.stdout: ], 2023-01-25T09:56:48.809 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 5, 2023-01-25T09:56:48.809 INFO:tasks.workunit.client.0.smithi101.stdout: "overlaps": [] 2023-01-25T09:56:48.809 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:48.810 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.810 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 5, 2023-01-25T09:56:48.810 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [ 2023-01-25T09:56:48.810 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.810 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 4, 2023-01-25T09:56:48.810 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap24" 2023-01-25T09:56:48.811 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:48.811 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.811 INFO:tasks.workunit.client.0.smithi101.stdout: "id": 5, 2023-01-25T09:56:48.811 INFO:tasks.workunit.client.0.smithi101.stdout: "name": "snap25" 2023-01-25T09:56:48.811 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:48.812 INFO:tasks.workunit.client.0.smithi101.stdout: ], 2023-01-25T09:56:48.812 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 6, 2023-01-25T09:56:48.812 INFO:tasks.workunit.client.0.smithi101.stdout: "overlaps": [] 2023-01-25T09:56:48.812 INFO:tasks.workunit.client.0.smithi101.stdout: }, 2023-01-25T09:56:48.812 INFO:tasks.workunit.client.0.smithi101.stdout: { 2023-01-25T09:56:48.813 INFO:tasks.workunit.client.0.smithi101.stdout: "id": "head", 2023-01-25T09:56:48.813 INFO:tasks.workunit.client.0.smithi101.stdout: "snapshots": [], 2023-01-25T09:56:48.813 INFO:tasks.workunit.client.0.smithi101.stdout: "size": 4 2023-01-25T09:56:48.813 INFO:tasks.workunit.client.0.smithi101.stdout: } 2023-01-25T09:56:48.813 INFO:tasks.workunit.client.0.smithi101.stdout: ] 2023-01-25T09:56:48.813 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:48.814 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2023-01-25T09:56:49.229 INFO:tasks.workunit.client.0.smithi101.stderr:nodeep-scrub is unset 2023-01-25T09:56:49.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2023-01-25T09:56:49.781 INFO:tasks.workunit.client.0.smithi101.stderr:noscrub is unset 2023-01-25T09:56:49.791 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-01-25T09:56:49.791 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2023-01-25T09:56:50.150 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2023-01-25T09:56:50.150 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T09:56:50.433 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T09:56:50.434 INFO:tasks.workunit.client.0.smithi101.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2023-01-25T09:56:47.792354+0000 25'7 37:63 [1,2,0] 1 [1,2,0] 1 25'7 2023-01-25T09:56:05.239579+0000 25'7 2023-01-25T09:56:05.239579+0000 0 1 periodic scrub scheduled @ 2023-01-26T09:56:05.239579+0000 4 0 2023-01-25T09:56:50.434 INFO:tasks.workunit.client.0.smithi101.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:48.795831+0000 0'0 38:45 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:50.434 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:47.792238+0000 0'0 37:35 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:50.434 INFO:tasks.workunit.client.0.smithi101.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2023-01-25T09:56:43.774174+0000 0'0 33:15 [0,1] 0 [0,1] 0 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:50.434 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:50.435 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T09:56:50.436 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T09:56:50.445 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2023-01-25T09:56:52.447 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-01-25T09:56:52.572 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:52.573 INFO:tasks.workunit.client.0.smithi101.stdout: "deep": true, 2023-01-25T09:56:52.573 INFO:tasks.workunit.client.0.smithi101.stdout: "stamp": "2023-01-18T09:55:12.568918+0000" 2023-01-25T09:56:52.573 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:52.583 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2023-01-25T09:56:57.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T09:56:57.877 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T09:56:57.878 INFO:tasks.workunit.client.0.smithi101.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2023-01-25T09:56:47.792354+0000 25'7 38:69 [1,2,0] 1 [1,2,0] 1 25'7 2023-01-18T09:55:12.568918+0000 25'7 2023-01-18T09:55:12.568918+0000 0 1 queued for deep scrub 4 0 2023-01-25T09:56:57.878 INFO:tasks.workunit.client.0.smithi101.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:48.795831+0000 0'0 39:47 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:57.878 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:47.792238+0000 0'0 38:36 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:57.878 INFO:tasks.workunit.client.0.smithi101.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:47.716995+0000 0'0 38:27 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:56:57.879 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:56:57.879 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T09:56:57.880 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T09:56:57.890 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:56:57.890 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T09:56:57.891 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T09:56:54.426+0000 7fecc3081700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-01-25T09:56:57.892 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2023-01-25T09:56:57.893 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T09:56:57.894 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2023-01-25T09:56:57.895 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2023-01-25T09:56:57.895 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2023-01-25T09:56:57.895 INFO:tasks.workunit.client.0.smithi101.stdout:prev count: 1 2023-01-25T09:56:57.896 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-01-25T09:56:58.009 INFO:tasks.workunit.client.0.smithi101.stdout:{ 2023-01-25T09:56:58.010 INFO:tasks.workunit.client.0.smithi101.stdout: "deep": true, 2023-01-25T09:56:58.010 INFO:tasks.workunit.client.0.smithi101.stdout: "stamp": "2023-01-18T09:55:18.005862+0000" 2023-01-25T09:56:58.010 INFO:tasks.workunit.client.0.smithi101.stdout:} 2023-01-25T09:56:58.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2023-01-25T09:57:03.021 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T09:57:03.313 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T09:57:03.314 INFO:tasks.workunit.client.0.smithi101.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2023-01-25T09:56:58.333261+0000 25'7 40:101 [1,2,0] 1 [1,2,0] 1 25'7 2023-01-25T09:56:58.333200+0000 25'7 2023-01-25T09:56:58.333200+0000 0 1 periodic scrub scheduled @ 2023-01-26T09:56:58.333200+0000 4 0 2023-01-25T09:57:03.314 INFO:tasks.workunit.client.0.smithi101.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:48.795831+0000 0'0 39:47 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:57:03.314 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:47.792238+0000 0'0 38:36 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:57:03.314 INFO:tasks.workunit.client.0.smithi101.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:56:47.716995+0000 0'0 38:27 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T09:55:55.751836+0000 0'0 2023-01-25T09:55:55.751836+0000 0 0 periodic scrub scheduled @ 2023-01-26T09:55:55.751836+0000 0 0 2023-01-25T09:57:03.314 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:57:03.315 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T09:57:03.315 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T09:57:03.325 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:57:03.326 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T09:57:03.327 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T09:56:54.426+0000 7fecc3081700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-01-25T09:57:03.327 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T09:57:03.328 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2023-01-25T09:57:03.329 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2023-01-25T09:57:03.329 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T09:57:03.329 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2023-01-25T09:57:03.330 INFO:tasks.workunit.client.0.smithi101.stdout:current count: 1 2023-01-25T09:57:03.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2023-01-25T09:57:03.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-01-25T09:57:03.331 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T09:57:03.331 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T09:57:03.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T09:57:03.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T09:57:03.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T09:57:03.637 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T09:57:03.637 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-01-25T09:57:03.637 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T09:57:03.637 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T09:57:03.640 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2023-01-25T09:57:03.640 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T09:57:04.924 INFO:tasks.workunit.client.0.smithi101.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:57:04.924 INFO:tasks.workunit.client.0.smithi101.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T09:57:04.924 INFO:tasks.workunit.client.0.smithi101.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T09:57:04.924 INFO:tasks.workunit.client.0.smithi101.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T09:57:04.925 INFO:tasks.workunit.client.0.smithi101.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:57:04.925 INFO:tasks.workunit.client.0.smithi101.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T09:57:04.926 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T09:57:04.926 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-01-25T09:57:04.926 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T09:57:04.926 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2023-01-25T09:57:05.966 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2023-01-25T09:57:05.966 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2023-01-25T09:57:05.967 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2023-01-25T09:57:05.967 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2023-01-25T09:57:05.967 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-01-25T09:57:05.967 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T09:57:05.968 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-01-25T09:57:05.968 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T09:57:05.968 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T09:57:05.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T09:57:05.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T09:57:05.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T09:57:06.075 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T09:57:06.075 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T09:57:06.076 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T09:57:06.076 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T09:57:06.077 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T09:57:06.077 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T09:57:06.078 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T09:57:06.079 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:57:06.080 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T09:57:06.080 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T09:57:06.081 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:57:06.081 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T09:57:06.082 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T09:57:06.082 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-01-25T09:57:06.098 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T09:57:06.098 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.098 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:57:06.099 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99476 2023-01-25T09:57:06.100 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T09:57:06.101 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T09:57:06.101 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: code=0 2023-01-25T09:57:06.101 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: main: teardown td/osd-mapper 0 2023-01-25T09:57:06.101 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-01-25T09:57:06.101 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-01-25T09:57:06.102 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-01-25T09:57:06.102 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T09:57:06.102 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T09:57:06.102 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T09:57:06.102 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T09:57:06.103 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T09:57:06.103 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T09:57:06.104 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T09:57:06.104 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T09:57:06.104 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T09:57:06.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T09:57:06.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T09:57:06.105 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T09:57:06.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:57:06.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T09:57:06.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T09:57:06.107 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:57:06.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T09:57:06.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-01-25T09:57:06.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-01-25T09:57:06.110 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T09:57:06.110 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.110 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99476 2023-01-25T09:57:06.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99476 2023-01-25T09:57:06.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T09:57:06.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T09:57:06.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: main: return 0 2023-01-25T09:57:06.113 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-25T09:57:06.113 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-25T09:57:06.184 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2023-01-25T09:57:06.184 DEBUG:teuthology.orchestra.run.smithi101:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=44e4095da12a8269eb6cfe163cbb0d00b1a6412c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2023-01-25T09:57:06.250 INFO:tasks.workunit.client.0.smithi101.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-01-25T09:57:06.259 INFO:tasks.workunit.client.0.smithi101.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-01-25T09:57:06.259 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-01-25T09:57:06.259 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-01-25T09:57:06.259 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: export PYTHONWARNINGS=ignore 2023-01-25T09:57:06.259 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: PYTHONWARNINGS=ignore 2023-01-25T09:57:06.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export CEPH_CONF=/dev/null 2023-01-25T09:57:06.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: CEPH_CONF=/dev/null 2023-01-25T09:57:06.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: unset CEPH_ARGS 2023-01-25T09:57:06.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: local code 2023-01-25T09:57:06.261 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: main: run td/osd-recovery-scrub 2023-01-25T09:57:06.261 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2023-01-25T09:57:06.261 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2023-01-25T09:57:06.261 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2023-01-25T09:57:06.261 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2023-01-25T09:57:06.262 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2023-01-25T09:57:06.262 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2023-01-25T09:57:06.262 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none ' 2023-01-25T09:57:06.262 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2023-01-25T09:57:06.262 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:28: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-01-25T09:57:06.263 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: set 2023-01-25T09:57:06.263 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-01-25T09:57:06.268 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: local 'funcs=TEST_recovery_scrub_1 2023-01-25T09:57:06.269 INFO:tasks.workunit.client.0.smithi101.stderr:TEST_recovery_scrub_2' 2023-01-25T09:57:06.269 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2023-01-25T09:57:06.270 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2023-01-25T09:57:06.270 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:38: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2023-01-25T09:57:06.270 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local poolname=test 2023-01-25T09:57:06.271 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:41: TEST_recovery_scrub_1: TESTDATA=testdata.115908 2023-01-25T09:57:06.271 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: OSDS=4 2023-01-25T09:57:06.272 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: PGS=1 2023-01-25T09:57:06.272 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: OBJECTS=100 2023-01-25T09:57:06.272 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: ERRORS=0 2023-01-25T09:57:06.273 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2023-01-25T09:57:06.273 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-01-25T09:57:06.273 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-01-25T09:57:06.274 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-01-25T09:57:06.274 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T09:57:06.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-01-25T09:57:06.275 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T09:57:06.275 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T09:57:06.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T09:57:06.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T09:57:06.277 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T09:57:06.277 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T09:57:06.277 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T09:57:06.278 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T09:57:06.278 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T09:57:06.278 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T09:57:06.279 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T09:57:06.279 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T09:57:06.279 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:57:06.280 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T09:57:06.280 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T09:57:06.281 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T09:57:06.281 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T09:57:06.281 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T09:57:06.282 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-01-25T09:57:06.287 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T09:57:06.287 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.287 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:06.288 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.115908 2023-01-25T09:57:06.288 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T09:57:06.289 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T09:57:06.289 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-01-25T09:57:06.289 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T09:57:06.290 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.290 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:06.290 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.115908 2023-01-25T09:57:06.291 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T09:57:06.291 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-01-25T09:57:06.291 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-01-25T09:57:06.292 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T09:57:06.292 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-01-25T09:57:06.293 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T09:57:06.293 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-01-25T09:57:06.293 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-25T09:57:06.293 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-25T09:57:06.294 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-25T09:57:06.294 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-01-25T09:57:06.294 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T09:57:06.611 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-25T09:57:06.611 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:57:06.612 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:57:06.612 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:57:06.612 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.612 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:06.613 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:57:06.613 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T09:57:06.862 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-25T09:57:06.864 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-25T09:57:06.864 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T09:57:06.864 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T09:57:06.864 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-25T09:57:06.865 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-25T09:57:06.865 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T09:57:06.865 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T09:57:06.865 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T09:57:06.865 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T09:57:06.866 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.866 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:06.866 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.115908/ceph-mon.a.asok 2023-01-25T09:57:06.866 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T09:57:06.867 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.115908/ceph-mon.a.asok config get fsid 2023-01-25T09:57:06.967 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-25T09:57:06.967 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T09:57:06.968 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T09:57:06.968 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-25T09:57:06.968 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-25T09:57:06.968 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T09:57:06.968 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T09:57:06.969 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T09:57:06.969 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T09:57:06.969 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:06.969 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:06.970 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.115908/ceph-mon.a.asok 2023-01-25T09:57:06.970 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T09:57:06.970 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.115908/ceph-mon.a.asok config get mon_host 2023-01-25T09:57:07.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2023-01-25T09:57:07.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-01-25T09:57:07.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-25T09:57:07.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-25T09:57:07.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-25T09:57:07.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-01-25T09:57:07.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-25T09:57:07.906 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-25T09:57:07.907 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:57:07.907 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:57:07.907 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:57:07.907 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:07.908 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:07.908 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:57:07.908 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T09:57:07.909 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T09:57:07.934 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: expr 4 - 1 2023-01-25T09:57:07.935 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: seq 0 3 2023-01-25T09:57:07.939 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T09:57:07.939 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2023-01-25T09:57:07.939 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T09:57:07.939 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T09:57:07.939 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-25T09:57:07.940 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T09:57:07.940 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-01-25T09:57:07.940 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T09:57:07.940 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:57:07.940 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:57:07.941 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:57:07.941 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-01-25T09:57:07.941 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-01-25T09:57:07.941 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T09:57:07.941 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T09:57:07.942 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T09:57:07.942 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T09:57:07.942 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:57:07.942 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:57:07.942 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:57:07.942 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:07.943 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:07.943 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:57:07.943 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:57:07.943 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:57:07.943 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T09:57:07.944 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T09:57:07.944 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T09:57:07.944 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T09:57:07.944 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:57:07.944 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:57:07.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:57:07.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T09:57:07.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T09:57:07.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-01-25T09:57:07.945 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T09:57:07.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=92b9b17f-c31a-48a8-8db4-4ba2ef82d1e1 2023-01-25T09:57:07.946 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 92b9b17f-c31a-48a8-8db4-4ba2ef82d1e1' 2023-01-25T09:57:07.946 INFO:tasks.workunit.client.0.smithi101.stdout:add osd0 92b9b17f-c31a-48a8-8db4-4ba2ef82d1e1 2023-01-25T09:57:07.946 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T09:57:07.956 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDz/NBj4lO0OBAArjVUF7NZohH1PYxybUZSpg== 2023-01-25T09:57:07.956 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDz/NBj4lO0OBAArjVUF7NZohH1PYxybUZSpg=="}' 2023-01-25T09:57:07.956 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 92b9b17f-c31a-48a8-8db4-4ba2ef82d1e1 -i td/osd-recovery-scrub/0/new.json 2023-01-25T09:57:08.287 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:57:08.298 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-01-25T09:57:08.299 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQDz/NBj4lO0OBAArjVUF7NZohH1PYxybUZSpg== --osd-uuid 92b9b17f-c31a-48a8-8db4-4ba2ef82d1e1 2023-01-25T09:57:08.326 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.320+0000 7fb2d702a580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:08.327 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.322+0000 7fb2d702a580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:08.328 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.323+0000 7fb2d702a580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:08.329 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.324+0000 7fb2d702a580 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T09:57:08.329 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.324+0000 7fb2d702a580 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T09:57:08.329 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.324+0000 7fb2d702a580 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T09:57:08.387 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:08.382+0000 7fb2d702a580 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-01-25T09:57:13.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-01-25T09:57:13.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T09:57:13.688 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-25T09:57:13.688 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd0 key to auth repository 2023-01-25T09:57:13.688 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T09:57:13.880 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-25T09:57:13.881 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.0 2023-01-25T09:57:13.881 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T09:57:13.882 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T09:57:13.882 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T09:57:13.882 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T09:57:13.904 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:13.899+0000 7fe1e7179580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:13.908 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:13.903+0000 7fe1e7179580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:13.909 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:13.904+0000 7fe1e7179580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:14.190 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-25T09:57:14.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:57:14.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T09:57:14.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:57:14.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:57:14.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:14.192 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:57:14.192 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:57:14.193 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:14.193 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:14.442 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:14.437+0000 7fe1e7179580 -1 Falling back to public interface 2023-01-25T09:57:14.532 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:15.533 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:15.533 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:15.534 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:57:15.534 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:57:15.534 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:15.535 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:15.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:16.762 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:16.757+0000 7fe1e7179580 -1 osd.0 0 log_to_monitors true 2023-01-25T09:57:16.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:16.840 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:16.840 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:57:16.841 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:57:16.841 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:16.841 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:17.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:18.192 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:18.192 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:18.193 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:57:18.193 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T09:57:18.193 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:18.194 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:18.503 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:19.504 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:19.820 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:20.821 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:20.821 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:20.822 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:57:20.822 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T09:57:20.823 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:20.823 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:21.132 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:22.133 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:22.134 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:22.134 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T09:57:22.134 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:57:22.135 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:22.135 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:22.488 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:23.490 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:23.490 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:23.490 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T09:57:23.491 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:57:23.491 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:23.492 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:23.805 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:24.806 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:24.807 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:24.807 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T09:57:24.807 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:57:24.808 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:24.808 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:25.118 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:26.119 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:26.119 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:26.119 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T09:57:26.120 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:57:26.120 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:26.121 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:26.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:27.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:27.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:27.468 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:57:27.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T09:57:27.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:27.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:28.031 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:29.032 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:29.033 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:29.033 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:57:29.034 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T09:57:29.034 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:29.034 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:29.493 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:30.494 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:30.494 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:30.495 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:57:30.495 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T09:57:30.496 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:30.496 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:30.903 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:31.904 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:31.905 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:31.905 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T09:57:31.905 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:57:31.906 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:31.906 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:32.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:33.211 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:33.211 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:33.212 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:57:33.212 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T09:57:33.213 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:33.213 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:33.901 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:34.902 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:34.903 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:34.903 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T09:57:34.903 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:57:34.904 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:34.904 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:35.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:36.586 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:36.587 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:36.587 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:57:36.588 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T09:57:36.589 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:36.589 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:37.018 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:38.019 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:38.019 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:38.019 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T09:57:38.020 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:57:38.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:38.021 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T09:57:38.328 INFO:tasks.workunit.client.0.smithi101.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/117048,v1:127.0.0.1:6803/117048] [v2:127.0.0.1:6804/117048,v1:127.0.0.1:6805/117048] exists,up 92b9b17f-c31a-48a8-8db4-4ba2ef82d1e1 2023-01-25T09:57:38.330 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:57:38.330 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:57:38.330 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:57:38.330 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T09:57:38.330 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2023-01-25T09:57:38.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T09:57:38.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T09:57:38.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-25T09:57:38.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T09:57:38.331 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-01-25T09:57:38.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T09:57:38.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:57:38.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:57:38.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:57:38.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-01-25T09:57:38.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-01-25T09:57:38.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T09:57:38.336 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T09:57:38.336 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T09:57:38.336 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T09:57:38.336 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:57:38.336 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:57:38.337 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:57:38.337 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:57:38.337 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:57:38.337 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:57:38.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:57:38.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:57:38.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T09:57:38.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T09:57:38.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T09:57:38.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T09:57:38.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:57:38.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:57:38.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:57:38.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T09:57:38.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T09:57:38.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-01-25T09:57:38.340 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T09:57:38.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=32aed829-7b2d-49e4-b93a-1bf6fd59cf3f 2023-01-25T09:57:38.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 32aed829-7b2d-49e4-b93a-1bf6fd59cf3f' 2023-01-25T09:57:38.340 INFO:tasks.workunit.client.0.smithi101.stdout:add osd1 32aed829-7b2d-49e4-b93a-1bf6fd59cf3f 2023-01-25T09:57:38.341 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T09:57:38.350 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAS/dBjLM2WFBAA4Ja8ptlTbVOD+UzQ03MxhQ== 2023-01-25T09:57:38.350 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAS/dBjLM2WFBAA4Ja8ptlTbVOD+UzQ03MxhQ=="}' 2023-01-25T09:57:38.350 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 32aed829-7b2d-49e4-b93a-1bf6fd59cf3f -i td/osd-recovery-scrub/1/new.json 2023-01-25T09:57:38.775 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:57:38.788 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-01-25T09:57:38.789 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQAS/dBjLM2WFBAA4Ja8ptlTbVOD+UzQ03MxhQ== --osd-uuid 32aed829-7b2d-49e4-b93a-1bf6fd59cf3f 2023-01-25T09:57:38.814 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.810+0000 7fd03daad580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:38.825 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.821+0000 7fd03daad580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:38.826 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.822+0000 7fd03daad580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:38.827 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.822+0000 7fd03daad580 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T09:57:38.827 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.822+0000 7fd03daad580 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T09:57:38.827 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.822+0000 7fd03daad580 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T09:57:38.902 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:38.898+0000 7fd03daad580 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-01-25T09:57:44.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-01-25T09:57:44.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T09:57:44.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-25T09:57:44.839 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd1 key to auth repository 2023-01-25T09:57:44.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T09:57:45.233 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.1 2023-01-25T09:57:45.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-25T09:57:45.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T09:57:45.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T09:57:45.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T09:57:45.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T09:57:45.257 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:45.253+0000 7f5eaf63f580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:45.259 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:45.255+0000 7f5eaf63f580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:45.260 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:45.256+0000 7f5eaf63f580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:57:45.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-25T09:57:45.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:57:45.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T09:57:45.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:57:45.606 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:57:45.606 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:45.606 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:57:45.606 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:57:45.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:45.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:45.779 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:45.775+0000 7f5eaf63f580 -1 Falling back to public interface 2023-01-25T09:57:45.951 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:46.952 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:46.953 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:46.953 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:57:46.953 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:57:46.954 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:46.954 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:47.264 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:48.260 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:57:48.255+0000 7f5eaf63f580 -1 osd.1 0 log_to_monitors true 2023-01-25T09:57:48.265 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:48.265 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:48.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:57:48.266 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:57:48.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:48.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:48.650 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:49.651 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:49.652 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:49.652 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T09:57:49.652 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:57:49.653 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:49.653 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:49.988 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:50.989 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:50.989 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:50.990 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T09:57:50.990 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:57:50.990 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:50.991 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:51.302 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:52.304 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:52.304 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:52.304 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T09:57:52.305 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:57:52.305 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:52.305 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:52.617 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:53.618 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:53.618 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:53.619 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T09:57:53.619 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:57:53.619 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:53.620 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:53.997 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:54.998 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:54.998 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:54.998 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T09:57:54.999 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:57:54.999 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:55.000 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:55.312 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:56.313 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:56.313 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:56.314 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:57:56.314 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T09:57:56.315 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:56.315 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:56.623 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:57.624 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:57.625 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:57.625 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T09:57:57.625 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:57:57.626 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:57.626 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:58.012 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:57:59.013 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:57:59.014 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:57:59.014 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T09:57:59.014 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:57:59.015 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:57:59.015 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:57:59.326 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:00.328 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:00.328 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:00.328 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T09:58:00.329 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:58:00.329 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:00.329 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:00.960 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:01.961 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:01.961 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:01.961 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T09:58:01.962 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:58:01.962 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:01.962 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:02.784 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:03.785 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:03.786 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:03.786 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:58:03.787 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T09:58:03.787 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:03.787 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:04.406 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:05.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:05.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:05.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T09:58:05.409 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:58:05.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:05.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:05.825 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:06.827 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:06.828 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:06.828 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T09:58:06.828 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:58:06.829 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:06.829 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:07.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:08.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:08.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:08.221 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:58:08.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T09:58:08.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:08.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:08.742 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:09.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:09.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:09.745 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:58:09.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T09:58:09.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:09.747 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T09:58:10.102 INFO:tasks.workunit.client.0.smithi101.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/118786,v1:127.0.0.1:6811/118786] [v2:127.0.0.1:6812/118786,v1:127.0.0.1:6813/118786] exists,up 32aed829-7b2d-49e4-b93a-1bf6fd59cf3f 2023-01-25T09:58:10.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:58:10.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:58:10.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:58:10.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T09:58:10.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2023-01-25T09:58:10.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T09:58:10.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T09:58:10.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-25T09:58:10.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T09:58:10.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-01-25T09:58:10.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T09:58:10.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:58:10.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:58:10.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:58:10.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-01-25T09:58:10.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-01-25T09:58:10.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T09:58:10.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T09:58:10.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T09:58:10.112 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T09:58:10.112 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:58:10.112 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:58:10.112 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:58:10.112 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:58:10.113 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:58:10.113 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:58:10.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:58:10.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:58:10.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T09:58:10.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T09:58:10.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T09:58:10.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T09:58:10.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:58:10.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:58:10.115 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:58:10.115 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T09:58:10.115 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T09:58:10.115 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-01-25T09:58:10.116 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T09:58:10.116 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f81a1ac4-e44c-4fa9-bef5-dcdb884bd449 2023-01-25T09:58:10.116 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 f81a1ac4-e44c-4fa9-bef5-dcdb884bd449' 2023-01-25T09:58:10.117 INFO:tasks.workunit.client.0.smithi101.stdout:add osd2 f81a1ac4-e44c-4fa9-bef5-dcdb884bd449 2023-01-25T09:58:10.117 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T09:58:10.122 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAy/dBj1MUJBxAAuZQQGodf2c6PXll5Qz+b1w== 2023-01-25T09:58:10.123 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAy/dBj1MUJBxAAuZQQGodf2c6PXll5Qz+b1w=="}' 2023-01-25T09:58:10.123 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f81a1ac4-e44c-4fa9-bef5-dcdb884bd449 -i td/osd-recovery-scrub/2/new.json 2023-01-25T09:58:10.534 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:58:10.546 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-01-25T09:58:10.547 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQAy/dBj1MUJBxAAuZQQGodf2c6PXll5Qz+b1w== --osd-uuid f81a1ac4-e44c-4fa9-bef5-dcdb884bd449 2023-01-25T09:58:10.573 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.568+0000 7ff9eccf1580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:10.584 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.579+0000 7ff9eccf1580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:10.584 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.580+0000 7ff9eccf1580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:10.585 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.581+0000 7ff9eccf1580 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T09:58:10.585 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.581+0000 7ff9eccf1580 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T09:58:10.586 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.581+0000 7ff9eccf1580 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T09:58:10.646 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:10.642+0000 7ff9eccf1580 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-01-25T09:58:16.415 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-01-25T09:58:16.415 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T09:58:16.416 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd2 key to auth repository 2023-01-25T09:58:16.417 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-25T09:58:16.417 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T09:58:16.803 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.2 2023-01-25T09:58:16.804 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-25T09:58:16.804 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T09:58:16.805 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T09:58:16.805 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T09:58:16.805 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T09:58:16.827 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:16.823+0000 7f10f18d5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:16.830 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:16.825+0000 7f10f18d5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:16.830 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:16.826+0000 7f10f18d5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:17.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-25T09:58:17.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:58:17.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T09:58:17.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:58:17.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:58:17.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:17.141 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:58:17.141 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:58:17.142 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:17.142 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:17.352 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:17.348+0000 7f10f18d5580 -1 Falling back to public interface 2023-01-25T09:58:17.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:18.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:18.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:18.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:58:18.470 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:58:18.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:18.471 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:18.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:19.610 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:19.605+0000 7f10f18d5580 -1 osd.2 0 log_to_monitors true 2023-01-25T09:58:19.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:19.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:19.777 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:58:19.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:58:19.778 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:19.778 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:20.138 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:21.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:21.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:21.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T09:58:21.140 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:58:21.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:21.141 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:21.452 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:22.454 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:22.454 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:22.455 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T09:58:22.455 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:58:22.456 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:22.456 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:22.766 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:23.767 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:23.768 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:23.768 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T09:58:23.768 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:58:23.769 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:23.769 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:24.103 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:25.104 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:25.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:25.105 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:58:25.105 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T09:58:25.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:25.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:25.416 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:26.417 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:26.417 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:26.418 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T09:58:26.418 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:58:26.418 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:26.419 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:26.742 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:27.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:27.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:27.744 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T09:58:27.745 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:58:27.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:27.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:28.085 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:29.086 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:29.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:29.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T09:58:29.088 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:58:29.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:29.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:29.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:30.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:30.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:30.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T09:58:30.409 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:58:30.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:30.410 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:30.718 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:31.720 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:31.720 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:31.720 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T09:58:31.721 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:58:31.721 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:31.722 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:32.373 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:33.375 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:33.375 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:33.376 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:58:33.376 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T09:58:33.377 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:33.377 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:33.751 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:34.753 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:34.753 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:34.753 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T09:58:34.754 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:58:34.754 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:34.754 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:35.478 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:36.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:36.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:36.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T09:58:36.481 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:58:36.481 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:36.481 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:36.835 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:37.836 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:37.836 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:37.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T09:58:37.837 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:58:37.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:37.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:38.554 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:39.555 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:39.555 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:39.555 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T09:58:39.556 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:58:39.556 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:39.557 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:40.148 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:41.149 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:41.150 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:41.150 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T09:58:41.150 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:58:41.151 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:41.151 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T09:58:41.458 INFO:tasks.workunit.client.0.smithi101.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/120532,v1:127.0.0.1:6819/120532] [v2:127.0.0.1:6820/120532,v1:127.0.0.1:6821/120532] exists,up f81a1ac4-e44c-4fa9-bef5-dcdb884bd449 2023-01-25T09:58:41.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:58:41.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:58:41.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:58:41.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T09:58:41.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2023-01-25T09:58:41.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T09:58:41.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T09:58:41.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-01-25T09:58:41.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T09:58:41.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-01-25T09:58:41.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T09:58:41.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T09:58:41.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T09:58:41.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T09:58:41.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-01-25T09:58:41.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-01-25T09:58:41.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T09:58:41.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T09:58:41.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T09:58:41.466 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T09:58:41.466 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T09:58:41.467 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T09:58:41.467 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T09:58:41.467 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T09:58:41.467 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T09:58:41.467 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:58:41.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T09:58:41.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T09:58:41.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T09:58:41.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T09:58:41.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T09:58:41.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T09:58:41.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T09:58:41.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T09:58:41.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T09:58:41.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T09:58:41.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T09:58:41.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-01-25T09:58:41.470 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T09:58:41.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=38db82df-535d-403c-b016-5470235fca9b 2023-01-25T09:58:41.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 38db82df-535d-403c-b016-5470235fca9b' 2023-01-25T09:58:41.471 INFO:tasks.workunit.client.0.smithi101.stdout:add osd3 38db82df-535d-403c-b016-5470235fca9b 2023-01-25T09:58:41.471 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T09:58:41.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBR/dBjhEtKHBAAfst8d3mwjGYUA6L59VN9MQ== 2023-01-25T09:58:41.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBR/dBjhEtKHBAAfst8d3mwjGYUA6L59VN9MQ=="}' 2023-01-25T09:58:41.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 38db82df-535d-403c-b016-5470235fca9b -i td/osd-recovery-scrub/3/new.json 2023-01-25T09:58:42.117 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:58:42.130 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-01-25T09:58:42.131 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQBR/dBjhEtKHBAAfst8d3mwjGYUA6L59VN9MQ== --osd-uuid 38db82df-535d-403c-b016-5470235fca9b 2023-01-25T09:58:42.157 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.151+0000 7f299126b580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:42.176 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.170+0000 7f299126b580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:42.176 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.171+0000 7f299126b580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:42.177 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.172+0000 7f299126b580 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T09:58:42.177 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.172+0000 7f299126b580 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T09:58:42.177 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.172+0000 7f299126b580 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T09:58:42.252 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:42.247+0000 7f299126b580 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-01-25T09:58:48.046 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-01-25T09:58:48.046 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T09:58:48.047 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd3 key to auth repository 2023-01-25T09:58:48.048 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2023-01-25T09:58:48.048 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T09:58:48.441 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2023-01-25T09:58:48.442 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.3 2023-01-25T09:58:48.442 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T09:58:48.443 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T09:58:48.443 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T09:58:48.443 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T09:58:48.465 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:48.460+0000 7f637b74c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:48.467 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:48.462+0000 7f637b74c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:48.468 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:48.463+0000 7f637b74c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T09:58:48.775 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2023-01-25T09:58:48.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T09:58:48.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=3 2023-01-25T09:58:48.776 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:48.984 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:48.979+0000 7f637b74c580 -1 Falling back to public interface 2023-01-25T09:58:49.122 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:50.123 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:50.124 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:50.124 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T09:58:50.125 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T09:58:50.125 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:50.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:50.433 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:51.285 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T09:58:51.281+0000 7f637b74c580 -1 osd.3 0 log_to_monitors true 2023-01-25T09:58:51.434 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:51.435 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:51.435 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T09:58:51.435 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T09:58:51.436 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:51.436 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:51.784 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:52.785 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:52.786 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:52.786 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T09:58:52.786 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T09:58:52.787 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:52.787 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:53.205 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:54.207 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:54.207 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:54.207 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T09:58:54.208 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T09:58:54.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:54.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:54.540 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:55.541 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:55.541 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:55.542 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T09:58:55.542 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T09:58:55.543 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:55.543 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:55.858 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:56.860 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:56.860 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:56.860 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T09:58:56.861 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T09:58:56.861 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:56.861 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:57.223 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:58.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:58.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:58.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T09:58:58.226 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T09:58:58.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:58.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:58:58.624 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:58:59.625 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:58:59.626 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:58:59.626 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T09:58:59.626 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T09:58:59.627 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:58:59.627 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:00.075 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:01.077 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:01.077 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:01.077 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T09:59:01.078 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T09:59:01.078 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:01.078 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:01.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:02.393 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:02.393 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:02.393 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T09:59:02.394 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T09:59:02.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:02.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:02.704 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:03.706 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:03.706 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:03.706 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T09:59:03.707 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T09:59:03.707 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:03.707 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:04.391 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:05.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:05.393 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:05.393 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T09:59:05.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T09:59:05.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:05.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:05.915 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:06.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:06.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:06.917 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T09:59:06.918 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T09:59:06.918 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:06.918 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:07.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:08.341 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:08.342 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:08.342 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T09:59:08.342 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T09:59:08.343 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:08.343 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:08.684 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:09.685 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:09.686 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:09.686 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T09:59:09.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T09:59:09.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:09.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:10.189 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:11.190 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:11.190 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:11.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T09:59:11.191 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T09:59:11.191 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:11.192 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:11.577 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:12.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:12.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:12.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T09:59:12.579 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T09:59:12.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:12.580 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:13.144 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T09:59:14.145 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T09:59:14.146 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T09:59:14.146 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T09:59:14.146 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T09:59:14.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T09:59:14.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T09:59:14.459 INFO:tasks.workunit.client.0.smithi101.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/122270,v1:127.0.0.1:6827/122270] [v2:127.0.0.1:6828/122270,v1:127.0.0.1:6829/122270] exists,up 38db82df-535d-403c-b016-5470235fca9b 2023-01-25T09:59:14.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T09:59:14.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T09:59:14.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T09:59:14.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: create_pool test 1 1 2023-01-25T09:59:14.461 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-01-25T09:59:14.943 INFO:tasks.workunit.client.0.smithi101.stderr:pool 'test' created 2023-01-25T09:59:14.953 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T09:59:15.954 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub_1: wait_for_clean 2023-01-25T09:59:15.955 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T09:59:15.955 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T09:59:15.955 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T09:59:15.955 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T09:59:15.955 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T09:59:15.956 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T09:59:15.957 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T09:59:15.957 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T09:59:15.957 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T09:59:15.957 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T09:59:16.056 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T09:59:16.056 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T09:59:16.057 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T09:59:16.057 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T09:59:16.057 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T09:59:16.356 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T09:59:16.357 INFO:tasks.workunit.client.0.smithi101.stderr:1 2023-01-25T09:59:16.357 INFO:tasks.workunit.client.0.smithi101.stderr:2 2023-01-25T09:59:16.357 INFO:tasks.workunit.client.0.smithi101.stderr:3' 2023-01-25T09:59:16.357 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T09:59:16.358 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T09:59:16.358 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T09:59:16.499 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836505 2023-01-25T09:59:16.499 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836505 2023-01-25T09:59:16.500 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505' 2023-01-25T09:59:16.500 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T09:59:16.500 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T09:59:16.621 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672979 2023-01-25T09:59:16.622 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672979 2023-01-25T09:59:16.622 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672979' 2023-01-25T09:59:16.622 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T09:59:16.622 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T09:59:16.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509449 2023-01-25T09:59:16.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509449 2023-01-25T09:59:16.747 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672979 2-64424509449' 2023-01-25T09:59:16.747 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T09:59:16.747 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T09:59:16.869 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345922 2023-01-25T09:59:16.869 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345922 2023-01-25T09:59:16.869 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672979 2-64424509449 3-85899345922' 2023-01-25T09:59:16.870 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T09:59:16.870 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836505 2023-01-25T09:59:16.870 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T09:59:16.871 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T09:59:16.871 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836505 2023-01-25T09:59:16.872 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T09:59:16.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836505 2023-01-25T09:59:16.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2023-01-25T09:59:16.875 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.0 seq 21474836505 2023-01-25T09:59:16.875 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T09:59:17.171 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836504 -lt 21474836505 2023-01-25T09:59:17.171 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-25T09:59:18.172 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-25T09:59:18.173 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T09:59:18.482 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836505 -lt 21474836505 2023-01-25T09:59:18.482 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T09:59:18.482 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672979 2023-01-25T09:59:18.483 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T09:59:18.483 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T09:59:18.484 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672979 2023-01-25T09:59:18.484 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T09:59:18.485 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672979 2023-01-25T09:59:18.485 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2023-01-25T09:59:18.485 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.1 seq 42949672979 2023-01-25T09:59:18.486 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T09:59:18.845 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672979 -lt 42949672979 2023-01-25T09:59:18.845 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T09:59:18.845 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509449 2023-01-25T09:59:18.845 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T09:59:18.846 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T09:59:18.846 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509449 2023-01-25T09:59:18.847 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T09:59:18.847 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509449 2023-01-25T09:59:18.847 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2023-01-25T09:59:18.848 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.2 seq 64424509449 2023-01-25T09:59:18.848 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T09:59:19.179 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509449 -lt 64424509449 2023-01-25T09:59:19.180 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T09:59:19.180 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345922 2023-01-25T09:59:19.181 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T09:59:19.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T09:59:19.182 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345922 2023-01-25T09:59:19.182 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T09:59:19.182 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345922 2023-01-25T09:59:19.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2023-01-25T09:59:19.183 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.3 seq 85899345922 2023-01-25T09:59:19.183 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T09:59:19.523 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345922 -lt 85899345922 2023-01-25T09:59:19.524 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T09:59:19.525 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T09:59:19.525 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T09:59:19.942 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 1 == 0 2023-01-25T09:59:19.942 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T09:59:19.943 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T09:59:19.943 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T09:59:19.943 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T09:59:19.943 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T09:59:19.943 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T09:59:19.944 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T09:59:20.255 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=1 2023-01-25T09:59:20.256 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T09:59:20.256 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T09:59:20.257 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T09:59:20.630 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 1 = 1 2023-01-25T09:59:20.630 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T09:59:20.630 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T09:59:20.631 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: ceph osd dump 2023-01-25T09:59:20.631 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2023-01-25T09:59:20.631 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: awk '{ print $2 }' 2023-01-25T09:59:20.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: poolid=1 2023-01-25T09:59:20.971 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: ceph pg dump pgs 2023-01-25T09:59:21.285 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T09:59:21.285 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T09:59:15.537732+0000 0'0 23:8 [1] 1 [1] 1 0'0 2023-01-25T09:59:14.800225+0000 0'0 2023-01-25T09:59:14.800225+0000 0 0 periodic scrub scheduled @ 2023-01-26T19:31:40.438873+0000 0 0 2023-01-25T09:59:21.286 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T09:59:21.286 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T09:59:21.288 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T09:59:21.297 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.115908 bs=1M count=50 2023-01-25T09:59:21.529 INFO:tasks.workunit.client.0.smithi101.stderr:50+0 records in 2023-01-25T09:59:21.529 INFO:tasks.workunit.client.0.smithi101.stderr:50+0 records out 2023-01-25T09:59:21.529 INFO:tasks.workunit.client.0.smithi101.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.23034 s, 228 MB/s 2023-01-25T09:59:21.530 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: seq 1 100 2023-01-25T09:59:21.530 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:21.530 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj1 testdata.115908 2023-01-25T09:59:23.004 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:23.004 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj2 testdata.115908 2023-01-25T09:59:24.568 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:24.568 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj3 testdata.115908 2023-01-25T09:59:26.049 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:26.049 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj4 testdata.115908 2023-01-25T09:59:27.530 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:27.531 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj5 testdata.115908 2023-01-25T09:59:29.153 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:29.153 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj6 testdata.115908 2023-01-25T09:59:30.676 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:30.677 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj7 testdata.115908 2023-01-25T09:59:32.032 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:32.032 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj8 testdata.115908 2023-01-25T09:59:33.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:33.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj9 testdata.115908 2023-01-25T09:59:35.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:35.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj10 testdata.115908 2023-01-25T09:59:36.575 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:36.576 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj11 testdata.115908 2023-01-25T09:59:38.006 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:38.006 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj12 testdata.115908 2023-01-25T09:59:39.562 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:39.562 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj13 testdata.115908 2023-01-25T09:59:41.052 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:41.052 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj14 testdata.115908 2023-01-25T09:59:42.574 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:42.575 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj15 testdata.115908 2023-01-25T09:59:44.373 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:44.373 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj16 testdata.115908 2023-01-25T09:59:45.870 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:45.870 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj17 testdata.115908 2023-01-25T09:59:47.447 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:47.448 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj18 testdata.115908 2023-01-25T09:59:48.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:48.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj19 testdata.115908 2023-01-25T09:59:50.459 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:50.459 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj20 testdata.115908 2023-01-25T09:59:51.865 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:51.865 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj21 testdata.115908 2023-01-25T09:59:53.379 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:53.379 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj22 testdata.115908 2023-01-25T09:59:55.243 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:55.244 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj23 testdata.115908 2023-01-25T09:59:56.733 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:56.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj24 testdata.115908 2023-01-25T09:59:58.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:58.240 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj25 testdata.115908 2023-01-25T09:59:59.661 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T09:59:59.662 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj26 testdata.115908 2023-01-25T10:00:01.251 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:01.252 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj27 testdata.115908 2023-01-25T10:00:02.816 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:02.816 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj28 testdata.115908 2023-01-25T10:00:04.389 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:04.389 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj29 testdata.115908 2023-01-25T10:00:05.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:05.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj30 testdata.115908 2023-01-25T10:00:07.525 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:07.526 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj31 testdata.115908 2023-01-25T10:00:09.199 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:09.199 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj32 testdata.115908 2023-01-25T10:00:10.863 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:10.864 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj33 testdata.115908 2023-01-25T10:00:12.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:12.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj34 testdata.115908 2023-01-25T10:00:14.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:14.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj35 testdata.115908 2023-01-25T10:00:15.815 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:15.815 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj36 testdata.115908 2023-01-25T10:00:17.304 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:17.305 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj37 testdata.115908 2023-01-25T10:00:18.827 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:18.827 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj38 testdata.115908 2023-01-25T10:00:20.484 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:20.484 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj39 testdata.115908 2023-01-25T10:00:21.998 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:21.998 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj40 testdata.115908 2023-01-25T10:00:23.521 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:23.521 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj41 testdata.115908 2023-01-25T10:00:25.086 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:25.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj42 testdata.115908 2023-01-25T10:00:26.620 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:26.620 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj43 testdata.115908 2023-01-25T10:00:28.018 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:28.018 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj44 testdata.115908 2023-01-25T10:00:29.567 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:29.567 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj45 testdata.115908 2023-01-25T10:00:31.166 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:31.166 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj46 testdata.115908 2023-01-25T10:00:32.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:32.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj47 testdata.115908 2023-01-25T10:00:34.313 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:34.313 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj48 testdata.115908 2023-01-25T10:00:35.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:35.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj49 testdata.115908 2023-01-25T10:00:37.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:37.477 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj50 testdata.115908 2023-01-25T10:00:39.100 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:39.100 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj51 testdata.115908 2023-01-25T10:00:40.757 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:40.758 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj52 testdata.115908 2023-01-25T10:00:42.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:42.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj53 testdata.115908 2023-01-25T10:00:44.046 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:44.046 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj54 testdata.115908 2023-01-25T10:00:45.636 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:45.636 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj55 testdata.115908 2023-01-25T10:00:47.419 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:47.419 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj56 testdata.115908 2023-01-25T10:00:49.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:49.127 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj57 testdata.115908 2023-01-25T10:00:50.792 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:50.792 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj58 testdata.115908 2023-01-25T10:00:52.299 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:52.299 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj59 testdata.115908 2023-01-25T10:00:53.738 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:53.738 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj60 testdata.115908 2023-01-25T10:00:55.312 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:55.313 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj61 testdata.115908 2023-01-25T10:00:56.861 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:56.862 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj62 testdata.115908 2023-01-25T10:00:58.435 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:00:58.435 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj63 testdata.115908 2023-01-25T10:01:00.009 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:00.009 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj64 testdata.115908 2023-01-25T10:01:01.691 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:01.691 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj65 testdata.115908 2023-01-25T10:01:03.248 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:03.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj66 testdata.115908 2023-01-25T10:01:04.847 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:04.847 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj67 testdata.115908 2023-01-25T10:01:06.379 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:06.380 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj68 testdata.115908 2023-01-25T10:01:07.769 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:07.769 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj69 testdata.115908 2023-01-25T10:01:09.293 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:09.293 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj70 testdata.115908 2023-01-25T10:01:10.975 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:10.976 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj71 testdata.115908 2023-01-25T10:01:12.524 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:12.524 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj72 testdata.115908 2023-01-25T10:01:13.964 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:13.964 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj73 testdata.115908 2023-01-25T10:01:15.429 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:15.429 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj74 testdata.115908 2023-01-25T10:01:17.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:17.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj75 testdata.115908 2023-01-25T10:01:18.693 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:18.694 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj76 testdata.115908 2023-01-25T10:01:20.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:20.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj77 testdata.115908 2023-01-25T10:01:21.641 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:21.641 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj78 testdata.115908 2023-01-25T10:01:23.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:23.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj79 testdata.115908 2023-01-25T10:01:24.696 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:24.697 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj80 testdata.115908 2023-01-25T10:01:26.212 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:26.212 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj81 testdata.115908 2023-01-25T10:01:27.704 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:27.704 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj82 testdata.115908 2023-01-25T10:01:29.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:29.261 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj83 testdata.115908 2023-01-25T10:01:30.826 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:30.827 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj84 testdata.115908 2023-01-25T10:01:32.417 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:32.417 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj85 testdata.115908 2023-01-25T10:01:33.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:33.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj86 testdata.115908 2023-01-25T10:01:35.355 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:35.356 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj87 testdata.115908 2023-01-25T10:01:36.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:36.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj88 testdata.115908 2023-01-25T10:01:38.503 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:38.504 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj89 testdata.115908 2023-01-25T10:01:39.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:39.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj90 testdata.115908 2023-01-25T10:01:41.550 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:41.551 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj91 testdata.115908 2023-01-25T10:01:43.224 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:43.224 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj92 testdata.115908 2023-01-25T10:01:44.831 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:44.831 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj93 testdata.115908 2023-01-25T10:01:46.397 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:46.397 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj94 testdata.115908 2023-01-25T10:01:48.045 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:48.045 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj95 testdata.115908 2023-01-25T10:01:49.527 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:49.527 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj96 testdata.115908 2023-01-25T10:01:51.118 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:51.119 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj97 testdata.115908 2023-01-25T10:01:52.699 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:52.700 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj98 testdata.115908 2023-01-25T10:01:54.258 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:54.258 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj99 testdata.115908 2023-01-25T10:01:55.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T10:01:55.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj100 testdata.115908 2023-01-25T10:01:57.237 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: rm -f testdata.115908 2023-01-25T10:01:57.244 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: ceph osd pool set test size 4 2023-01-25T10:01:57.672 INFO:tasks.workunit.client.0.smithi101.stderr:set pool 1 size to 4 2023-01-25T10:01:57.682 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:73: TEST_recovery_scrub_1: set -o pipefail 2023-01-25T10:01:57.683 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: count=0 2023-01-25T10:01:57.683 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T10:01:57.683 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T10:01:57.684 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T10:01:57.684 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T10:01:58.006 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T10:01:58.016 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2023-01-25T10:02:00.017 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 0 -eq 10 2023-01-25T10:02:00.018 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 0 + 1 2023-01-25T10:02:00.018 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=1 2023-01-25T10:02:00.019 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T10:02:00.019 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T10:02:00.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T10:02:00.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T10:02:00.318 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T10:02:00.328 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2023-01-25T10:02:02.329 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 1 -eq 10 2023-01-25T10:02:02.330 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 1 + 1 2023-01-25T10:02:02.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=2 2023-01-25T10:02:02.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T10:02:02.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T10:02:02.332 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T10:02:02.333 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T10:02:02.742 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T10:02:02.752 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:80: TEST_recovery_scrub_1: break 2023-01-25T10:02:02.753 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:90: TEST_recovery_scrub_1: set +o pipefail 2023-01-25T10:02:02.753 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:91: TEST_recovery_scrub_1: ceph pg dump pgs 2023-01-25T10:02:03.035 INFO:tasks.workunit.client.0.smithi101.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T10:02:03.036 INFO:tasks.workunit.client.0.smithi101.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2023-01-25T10:01:59.301933+0000 24'1300 27:1327 [1,0,2,3] 1 [1,0] 1 0'0 2023-01-25T09:59:14.800225+0000 0'0 2023-01-25T09:59:14.800225+0000 0 0 periodic scrub scheduled @ 2023-01-26T13:14:21.131021+0000 0 0 2023-01-25T10:02:03.036 INFO:tasks.workunit.client.0.smithi101.stdout: 2023-01-25T10:02:03.037 INFO:tasks.workunit.client.0.smithi101.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T10:02:03.037 INFO:tasks.workunit.client.0.smithi101.stderr:dumped pgs 2023-01-25T10:02:03.047 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:93: TEST_recovery_scrub_1: sleep 10 2023-01-25T10:02:13.048 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2023-01-25T10:02:13.049 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T10:02:13.050 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T10:02:13.050 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T10:02:13.050 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T10:02:13.050 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T10:02:18.366 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T10:02:18.366 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: declare -a err_strings 2023-01-25T10:02:18.366 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:98: TEST_recovery_scrub_1: err_strings[0]='not scheduling scrubs due to active recovery' 2023-01-25T10:02:18.366 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: expr 4 - 1 2023-01-25T10:02:18.367 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: seq 0 3 2023-01-25T10:02:18.367 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.368 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2023-01-25T10:02:18.370 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:03.355+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.370 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:04.313+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.370 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:06.260+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.370 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:08.274+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.371 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:09.227+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.371 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:11.198+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.371 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:15.144+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.372 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:16.170+0000 7fe1d6101700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.372 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.372 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2023-01-25T10:02:18.383 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:00.018+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.384 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:00.986+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.384 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:01.967+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.384 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:04.885+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.384 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:11.891+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.384 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:14.921+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.385 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:15.933+0000 7f5e9e5c7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.385 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.385 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2023-01-25T10:02:18.386 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:01:59.996+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.387 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:04.002+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.387 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:04.972+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.387 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:09.082+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.387 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:12.124+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.387 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:15.050+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.388 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:17.069+0000 7f10e085d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.388 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.388 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2023-01-25T10:02:18.389 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:01.846+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.389 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:02.853+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.389 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:04.892+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.390 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:06.908+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.390 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:09.921+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.390 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:12.910+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.390 INFO:tasks.workunit.client.0.smithi101.stdout:2023-01-25T10:02:16.952+0000 7f636a6d4700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T10:02:18.391 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2023-01-25T10:02:18.391 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: found=false 2023-01-25T10:02:18.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:107: TEST_recovery_scrub_1: count=0 2023-01-25T10:02:18.392 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: expr 4 - 1 2023-01-25T10:02:18.392 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: seq 0 3 2023-01-25T10:02:18.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.393 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2023-01-25T10:02:18.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T10:02:18.394 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 0 + 1 2023-01-25T10:02:18.395 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=1 2023-01-25T10:02:18.395 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.395 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2023-01-25T10:02:18.405 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T10:02:18.405 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 1 + 1 2023-01-25T10:02:18.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=2 2023-01-25T10:02:18.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2023-01-25T10:02:18.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T10:02:18.409 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 2 + 1 2023-01-25T10:02:18.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=3 2023-01-25T10:02:18.410 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:18.411 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2023-01-25T10:02:18.412 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T10:02:18.412 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 3 + 1 2023-01-25T10:02:18.413 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=4 2023-01-25T10:02:18.414 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: '[' true = false ']' 2023-01-25T10:02:18.414 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2023-01-25T10:02:18.414 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:123: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2023-01-25T10:02:18.414 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-01-25T10:02:18.414 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T10:02:18.415 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-01-25T10:02:18.415 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T10:02:18.415 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T10:02:18.415 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T10:02:18.415 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T10:02:18.416 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T10:02:18.419 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T10:02:18.420 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T10:02:18.421 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T10:02:18.421 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T10:02:18.423 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T10:02:18.423 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T10:02:18.423 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T10:02:18.424 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:02:18.424 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T10:02:18.425 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T10:02:18.425 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:02:18.425 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T10:02:18.426 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T10:02:18.426 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-01-25T10:02:18.456 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T10:02:18.457 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:18.457 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:18.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.115908 2023-01-25T10:02:18.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T10:02:18.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T10:02:18.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:125: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2023-01-25T10:02:18.459 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:131: TEST_recovery_scrub_1: echo 'TEST PASSED' 2023-01-25T10:02:18.459 INFO:tasks.workunit.client.0.smithi101.stdout:TEST PASSED 2023-01-25T10:02:18.461 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:132: TEST_recovery_scrub_1: return 0 2023-01-25T10:02:18.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2023-01-25T10:02:18.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2023-01-25T10:02:18.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:230: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2023-01-25T10:02:18.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:231: TEST_recovery_scrub_2: local poolname=test 2023-01-25T10:02:18.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:233: TEST_recovery_scrub_2: TESTDATA=testdata.115908 2023-01-25T10:02:18.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: OSDS=8 2023-01-25T10:02:18.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: PGS=32 2023-01-25T10:02:18.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:236: TEST_recovery_scrub_2: OBJECTS=40 2023-01-25T10:02:18.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2023-01-25T10:02:18.463 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-01-25T10:02:18.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-01-25T10:02:18.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-01-25T10:02:18.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T10:02:18.464 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-01-25T10:02:18.464 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T10:02:18.465 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T10:02:18.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T10:02:18.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T10:02:18.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T10:02:18.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T10:02:18.466 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T10:02:18.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T10:02:18.466 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T10:02:18.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T10:02:18.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T10:02:18.468 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T10:02:18.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:02:18.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T10:02:18.468 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T10:02:18.469 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:02:18.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T10:02:18.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T10:02:18.469 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-01-25T10:02:18.469 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T10:02:18.470 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:18.470 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:18.470 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.115908 2023-01-25T10:02:18.472 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T10:02:18.472 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T10:02:18.472 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-01-25T10:02:18.473 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T10:02:18.473 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:18.473 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:18.473 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.115908 2023-01-25T10:02:18.473 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T10:02:18.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-01-25T10:02:18.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T10:02:18.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-01-25T10:02:18.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T10:02:18.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-01-25T10:02:18.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-25T10:02:18.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-25T10:02:18.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-25T10:02:18.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-01-25T10:02:18.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T10:02:18.834 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-25T10:02:18.834 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:02:18.834 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:02:18.834 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:02:18.835 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:18.835 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:18.835 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:02:18.835 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T10:02:19.127 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-25T10:02:19.129 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-25T10:02:19.129 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T10:02:19.130 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T10:02:19.130 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-25T10:02:19.130 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-25T10:02:19.130 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T10:02:19.130 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T10:02:19.131 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T10:02:19.131 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T10:02:19.131 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:19.131 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:19.132 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.115908/ceph-mon.a.asok 2023-01-25T10:02:19.132 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T10:02:19.132 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.115908/ceph-mon.a.asok config get fsid 2023-01-25T10:02:19.233 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-25T10:02:19.233 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T10:02:19.234 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T10:02:19.234 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-25T10:02:19.234 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-25T10:02:19.235 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T10:02:19.235 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T10:02:19.235 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T10:02:19.235 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T10:02:19.235 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:19.236 INFO:tasks.workunit.client.0.smithi101.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:19.236 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.115908/ceph-mon.a.asok 2023-01-25T10:02:19.236 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T10:02:19.236 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.115908/ceph-mon.a.asok config get mon_host 2023-01-25T10:02:19.334 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:241: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2023-01-25T10:02:19.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-01-25T10:02:19.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-25T10:02:19.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-25T10:02:19.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-25T10:02:19.336 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-01-25T10:02:19.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-25T10:02:20.182 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-25T10:02:20.182 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:02:20.182 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:02:20.183 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:02:20.183 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:20.183 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:20.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:02:20.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T10:02:20.184 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T10:02:20.208 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: expr 8 - 1 2023-01-25T10:02:20.209 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: seq 0 7 2023-01-25T10:02:20.213 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:20.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:02:20.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:02:20.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:02:20.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-25T10:02:20.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:02:20.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-01-25T10:02:20.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:02:20.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:02:20.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:02:20.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:02:20.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-01-25T10:02:20.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-01-25T10:02:20.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:02:20.217 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:02:20.217 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:02:20.217 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:02:20.217 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:02:20.218 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:02:20.218 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:02:20.218 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:20.218 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:20.218 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:02:20.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:02:20.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:02:20.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:02:20.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:02:20.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:02:20.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:02:20.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:02:20.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:02:20.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:02:20.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:02:20.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:02:20.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-01-25T10:02:20.221 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:02:20.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0b91b4e7-029e-40ac-a523-3bac1a926aaf 2023-01-25T10:02:20.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 0b91b4e7-029e-40ac-a523-3bac1a926aaf' 2023-01-25T10:02:20.222 INFO:tasks.workunit.client.0.smithi101.stdout:add osd0 0b91b4e7-029e-40ac-a523-3bac1a926aaf 2023-01-25T10:02:20.222 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:02:20.231 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAs/tBjHOF7DRAA10HLxXbPt/wwETyMOxwyuA== 2023-01-25T10:02:20.231 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAs/tBjHOF7DRAA10HLxXbPt/wwETyMOxwyuA=="}' 2023-01-25T10:02:20.231 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0b91b4e7-029e-40ac-a523-3bac1a926aaf -i td/osd-recovery-scrub/0/new.json 2023-01-25T10:02:20.536 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:02:20.547 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-01-25T10:02:20.548 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAs/tBjHOF7DRAA10HLxXbPt/wwETyMOxwyuA== --osd-uuid 0b91b4e7-029e-40ac-a523-3bac1a926aaf 2023-01-25T10:02:20.575 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.570+0000 7f05be358580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:20.577 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.572+0000 7f05be358580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:20.578 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.573+0000 7f05be358580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:20.579 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.574+0000 7f05be358580 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T10:02:20.579 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.574+0000 7f05be358580 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T10:02:20.579 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.574+0000 7f05be358580 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T10:02:20.636 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:20.631+0000 7f05be358580 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-01-25T10:02:26.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-01-25T10:02:26.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:02:26.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-25T10:02:26.338 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd0 key to auth repository 2023-01-25T10:02:26.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:02:26.733 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-25T10:02:26.733 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.0 2023-01-25T10:02:26.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:02:26.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:02:26.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:02:26.735 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:02:26.758 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:26.752+0000 7f3409ba3580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:26.760 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:26.755+0000 7f3409ba3580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:26.761 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:26.756+0000 7f3409ba3580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:27.067 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-25T10:02:27.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:02:27.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T10:02:27.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:02:27.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:02:27.069 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:27.070 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:02:27.070 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:02:27.071 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:27.072 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:27.276 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:27.271+0000 7f3409ba3580 -1 Falling back to public interface 2023-01-25T10:02:27.406 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:28.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:28.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:28.407 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:02:28.408 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:02:28.408 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:28.409 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:28.716 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:29.657 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:29.652+0000 7f3409ba3580 -1 osd.0 0 log_to_monitors true 2023-01-25T10:02:29.717 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:29.717 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:29.718 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:02:29.718 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:02:29.719 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:29.719 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:30.055 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:31.057 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:31.057 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:31.057 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:02:31.058 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:02:31.058 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:31.059 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:31.370 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:32.370 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:32.370 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:32.371 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:02:32.371 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:02:32.372 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:32.372 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:32.684 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:33.685 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:33.686 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:33.686 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:02:33.686 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:02:33.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:33.687 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:33.994 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:34.996 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:34.996 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:34.996 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:02:34.997 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:02:34.997 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:34.997 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:35.365 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:36.367 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:36.367 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:36.367 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:02:36.368 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:02:36.368 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:36.369 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:36.678 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:37.680 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:37.680 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:37.680 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:02:37.681 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:02:37.681 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:37.681 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:37.991 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:38.992 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:38.993 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:38.993 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:02:38.993 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:02:38.994 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:38.994 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:39.394 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:40.395 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:40.396 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:40.396 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:02:40.396 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:02:40.397 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:40.397 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:40.940 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:41.942 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:41.943 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:41.943 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:02:41.944 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:02:41.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:41.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:42.486 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:43.488 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:43.488 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:43.489 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:02:43.490 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:02:43.490 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:43.491 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:44.082 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:45.083 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:45.083 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:45.083 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:02:45.084 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:02:45.085 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:45.085 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:45.583 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:46.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:46.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:46.586 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:02:46.587 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:02:46.587 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:46.587 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:47.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:48.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:48.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:48.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:02:48.338 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:02:48.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:48.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:49.356 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:50.362 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:50.362 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:50.362 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:02:50.362 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:02:50.363 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:50.363 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:50.789 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:02:51.791 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:02:51.791 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:51.791 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:02:51.792 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:02:51.792 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:51.792 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T10:02:52.102 INFO:tasks.workunit.client.0.smithi101.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127118,v1:127.0.0.1:6803/127118] [v2:127.0.0.1:6804/127118,v1:127.0.0.1:6805/127118] exists,up 0b91b4e7-029e-40ac-a523-3bac1a926aaf 2023-01-25T10:02:52.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:02:52.106 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:02:52.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:02:52.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:02:52.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:02:52.107 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:02:52.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:02:52.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-25T10:02:52.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:02:52.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-01-25T10:02:52.108 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:02:52.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:02:52.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:02:52.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:02:52.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-01-25T10:02:52.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-01-25T10:02:52.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:02:52.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:02:52.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:02:52.110 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:02:52.110 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:02:52.110 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:02:52.111 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:02:52.111 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:02:52.111 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:02:52.111 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:02:52.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:02:52.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:02:52.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:02:52.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:02:52.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:02:52.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:02:52.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:02:52.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:02:52.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:02:52.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:02:52.113 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:02:52.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-01-25T10:02:52.114 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:02:52.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0ed9e5fe-5a05-4b5e-93ba-29aac5ff1fa9 2023-01-25T10:02:52.114 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 0ed9e5fe-5a05-4b5e-93ba-29aac5ff1fa9' 2023-01-25T10:02:52.115 INFO:tasks.workunit.client.0.smithi101.stdout:add osd1 0ed9e5fe-5a05-4b5e-93ba-29aac5ff1fa9 2023-01-25T10:02:52.115 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:02:52.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBM/tBjxO0SBxAAXQonfzcpzAGSDe6xp9ls3g== 2023-01-25T10:02:52.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBM/tBjxO0SBxAAXQonfzcpzAGSDe6xp9ls3g=="}' 2023-01-25T10:02:52.126 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0ed9e5fe-5a05-4b5e-93ba-29aac5ff1fa9 -i td/osd-recovery-scrub/1/new.json 2023-01-25T10:02:52.535 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:02:52.548 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-01-25T10:02:52.549 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBM/tBjxO0SBxAAXQonfzcpzAGSDe6xp9ls3g== --osd-uuid 0ed9e5fe-5a05-4b5e-93ba-29aac5ff1fa9 2023-01-25T10:02:52.576 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.571+0000 7f285fd58580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:52.594 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.590+0000 7f285fd58580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:52.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.591+0000 7f285fd58580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:52.595 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.591+0000 7f285fd58580 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T10:02:52.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.591+0000 7f285fd58580 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T10:02:52.596 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.591+0000 7f285fd58580 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T10:02:52.671 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:52.666+0000 7f285fd58580 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-01-25T10:02:58.489 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-01-25T10:02:58.489 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:02:58.489 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-25T10:02:58.490 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd1 key to auth repository 2023-01-25T10:02:58.490 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:02:58.880 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-25T10:02:58.880 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.1 2023-01-25T10:02:58.881 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:02:58.881 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:02:58.881 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:02:58.882 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:02:58.905 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:58.899+0000 7f6056658580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:58.906 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:58.901+0000 7f6056658580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:58.907 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:58.902+0000 7f6056658580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:02:59.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-25T10:02:59.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:02:59.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T10:02:59.209 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:02:59.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:02:59.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:02:59.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:02:59.210 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:02:59.211 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:02:59.211 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:02:59.431 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:02:59.426+0000 7f6056658580 -1 Falling back to public interface 2023-01-25T10:02:59.550 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:00.551 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:00.551 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:00.552 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:03:00.552 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:03:00.553 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:00.553 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:00.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:01.776 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:01.771+0000 7f6056658580 -1 osd.1 0 log_to_monitors true 2023-01-25T10:03:01.855 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:01.855 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:01.856 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:03:01.857 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:03:01.857 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:01.857 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:02.246 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:03.247 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:03.247 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:03.248 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:03:03.248 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:03:03.248 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:03.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:03.576 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:04.577 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:04.577 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:04.578 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:03:04.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:03:04.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:04.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:04.889 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:05.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:05.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:05.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:03:05.894 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:03:05.894 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:05.895 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:06.223 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:07.224 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:07.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:07.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:03:07.225 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:03:07.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:07.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:07.557 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:08.558 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:08.559 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:08.560 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:03:08.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:03:08.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:08.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:08.865 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:09.866 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:09.866 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:09.867 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:03:09.867 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:03:09.868 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:09.868 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:10.204 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:11.205 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:11.205 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:11.205 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:03:11.206 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:03:11.206 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:11.207 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:11.564 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:12.566 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:12.566 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:12.567 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:03:12.568 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:03:12.568 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:12.569 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:12.870 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:13.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:13.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:13.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:03:13.873 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:03:13.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:13.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:14.426 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:15.428 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:15.428 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:15.428 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:03:15.429 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:03:15.429 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:15.429 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:15.746 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:16.748 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:16.748 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:16.749 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:03:16.749 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:03:16.750 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:16.750 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:17.058 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:18.059 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:18.059 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:18.060 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:03:18.060 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:03:18.061 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:18.061 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:18.754 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:19.755 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:19.755 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:19.756 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:03:19.756 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:03:19.757 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:19.757 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:20.342 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:21.343 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:21.343 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:21.344 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:03:21.344 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:03:21.345 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:21.345 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:21.727 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:22.729 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:22.729 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:22.729 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:03:22.730 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:03:22.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:22.730 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:23.155 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:24.156 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:24.156 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:24.157 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:03:24.157 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:03:24.158 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:24.158 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T10:03:24.469 INFO:tasks.workunit.client.0.smithi101.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/128857,v1:127.0.0.1:6811/128857] [v2:127.0.0.1:6812/128857,v1:127.0.0.1:6813/128857] exists,up 0ed9e5fe-5a05-4b5e-93ba-29aac5ff1fa9 2023-01-25T10:03:24.472 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:03:24.473 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:03:24.473 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:03:24.473 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:03:24.473 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:03:24.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:03:24.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:03:24.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-25T10:03:24.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:03:24.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-01-25T10:03:24.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:03:24.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:03:24.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:03:24.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:03:24.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-01-25T10:03:24.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-01-25T10:03:24.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:03:24.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:03:24.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:03:24.477 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:03:24.477 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:03:24.477 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:03:24.477 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:03:24.477 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:03:24.478 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:03:24.478 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:03:24.478 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:03:24.478 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:03:24.478 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:03:24.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:03:24.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:03:24.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:03:24.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:03:24.479 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:03:24.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:03:24.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:03:24.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:03:24.480 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-01-25T10:03:24.480 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:03:24.481 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=6dfa4d9b-6c9d-4fc3-bb6e-51dd5c05e90f 2023-01-25T10:03:24.481 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 6dfa4d9b-6c9d-4fc3-bb6e-51dd5c05e90f' 2023-01-25T10:03:24.481 INFO:tasks.workunit.client.0.smithi101.stdout:add osd2 6dfa4d9b-6c9d-4fc3-bb6e-51dd5c05e90f 2023-01-25T10:03:24.482 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:03:24.489 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBs/tBjPqvlHBAAn7x7nvNrhOojQq6cjj83Pw== 2023-01-25T10:03:24.489 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBs/tBjPqvlHBAAn7x7nvNrhOojQq6cjj83Pw=="}' 2023-01-25T10:03:24.489 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 6dfa4d9b-6c9d-4fc3-bb6e-51dd5c05e90f -i td/osd-recovery-scrub/2/new.json 2023-01-25T10:03:24.990 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:03:25.002 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-01-25T10:03:25.003 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBs/tBjPqvlHBAAn7x7nvNrhOojQq6cjj83Pw== --osd-uuid 6dfa4d9b-6c9d-4fc3-bb6e-51dd5c05e90f 2023-01-25T10:03:25.029 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.025+0000 7f7439c1c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:25.052 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.044+0000 7f7439c1c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:25.053 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.049+0000 7f7439c1c580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:25.054 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.050+0000 7f7439c1c580 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T10:03:25.054 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.050+0000 7f7439c1c580 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T10:03:25.054 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.050+0000 7f7439c1c580 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T10:03:25.148 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:25.143+0000 7f7439c1c580 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-01-25T10:03:31.071 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-01-25T10:03:31.071 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:03:31.072 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-25T10:03:31.072 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd2 key to auth repository 2023-01-25T10:03:31.073 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:03:31.465 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-25T10:03:31.465 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.2 2023-01-25T10:03:31.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:03:31.466 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:03:31.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:03:31.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:03:31.490 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:31.485+0000 7fca475b5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:31.492 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:31.488+0000 7fca475b5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:31.492 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:31.489+0000 7fca475b5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:31.804 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-25T10:03:31.805 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:03:31.805 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T10:03:31.806 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:03:31.806 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:03:31.806 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:31.806 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:03:31.808 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:03:31.808 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:31.808 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:32.013 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:32.009+0000 7fca475b5580 -1 Falling back to public interface 2023-01-25T10:03:32.144 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:33.145 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:33.146 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:33.146 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:03:33.146 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:03:33.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:33.148 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:33.455 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:34.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:34.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:34.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:03:34.458 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:03:34.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:34.459 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:34.459 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:34.454+0000 7fca475b5580 -1 osd.2 0 log_to_monitors true 2023-01-25T10:03:34.768 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:35.769 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:35.769 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:35.770 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:03:35.770 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:03:35.770 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:35.771 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:36.146 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:37.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:37.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:37.148 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:03:37.148 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:03:37.148 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:37.149 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:37.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:38.459 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:38.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:38.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:03:38.461 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:03:38.461 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:38.461 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:38.771 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:39.772 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:39.773 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:39.773 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:03:39.773 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:03:39.774 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:39.774 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:40.136 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:41.137 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:41.138 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:41.138 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:03:41.138 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:03:41.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:41.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:41.497 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:42.498 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:42.498 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:42.498 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:03:42.499 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:03:42.499 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:42.499 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:42.834 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:43.835 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:43.836 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:43.836 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:03:43.836 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:03:43.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:43.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:44.147 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:45.148 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:45.149 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:45.150 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:03:45.150 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:03:45.151 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:45.151 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:45.450 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:46.451 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:46.452 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:46.452 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:03:46.452 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:03:46.453 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:46.453 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:46.858 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:47.860 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:47.860 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:47.860 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:03:47.861 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:03:47.861 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:47.861 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:48.319 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:49.320 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:49.320 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:49.321 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:03:49.321 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:03:49.322 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:49.322 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:49.759 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:50.761 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:50.761 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:50.762 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:03:50.762 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:03:50.763 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:50.763 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:51.129 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:52.130 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:52.131 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:52.131 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:03:52.132 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:03:52.132 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:52.132 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:52.588 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:53.589 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:53.590 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:53.590 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:03:53.591 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:03:53.591 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:53.591 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:53.996 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:54.997 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:54.997 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:54.998 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:03:54.998 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:03:54.999 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:54.999 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:55.502 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:56.503 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:56.504 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:56.504 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:03:56.504 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:03:56.505 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:56.505 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:56.848 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:03:57.849 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:03:57.849 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:03:57.850 INFO:tasks.workunit.client.0.smithi101.stdout:19 2023-01-25T10:03:57.850 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T10:03:57.851 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:03:57.851 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T10:03:58.176 INFO:tasks.workunit.client.0.smithi101.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/130628,v1:127.0.0.1:6819/130628] [v2:127.0.0.1:6820/130628,v1:127.0.0.1:6821/130628] exists,up 6dfa4d9b-6c9d-4fc3-bb6e-51dd5c05e90f 2023-01-25T10:03:58.180 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:03:58.180 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:03:58.180 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:03:58.180 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:03:58.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:03:58.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:03:58.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:03:58.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-01-25T10:03:58.181 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:03:58.182 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-01-25T10:03:58.182 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:03:58.182 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:03:58.182 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:03:58.182 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:03:58.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-01-25T10:03:58.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-01-25T10:03:58.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:03:58.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:03:58.183 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:03:58.183 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:03:58.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:03:58.184 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:03:58.184 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:03:58.185 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:03:58.185 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:03:58.185 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:03:58.185 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:03:58.185 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:03:58.186 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:03:58.186 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:03:58.186 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:03:58.186 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:03:58.186 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:03:58.187 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:03:58.187 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:03:58.187 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:03:58.187 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:03:58.187 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-01-25T10:03:58.187 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:03:58.188 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bd097d5c-d859-4230-a678-81b9c511959e 2023-01-25T10:03:58.188 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 bd097d5c-d859-4230-a678-81b9c511959e' 2023-01-25T10:03:58.188 INFO:tasks.workunit.client.0.smithi101.stdout:add osd3 bd097d5c-d859-4230-a678-81b9c511959e 2023-01-25T10:03:58.189 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:03:58.197 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCO/tBjTp11CxAA19I+rsWH36Pd8dv+GxURyA== 2023-01-25T10:03:58.197 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCO/tBjTp11CxAA19I+rsWH36Pd8dv+GxURyA=="}' 2023-01-25T10:03:58.197 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bd097d5c-d859-4230-a678-81b9c511959e -i td/osd-recovery-scrub/3/new.json 2023-01-25T10:03:58.617 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:03:58.629 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-01-25T10:03:58.630 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQCO/tBjTp11CxAA19I+rsWH36Pd8dv+GxURyA== --osd-uuid bd097d5c-d859-4230-a678-81b9c511959e 2023-01-25T10:03:58.656 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.651+0000 7fcc1609a580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:58.666 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.661+0000 7fcc1609a580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:58.667 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.662+0000 7fcc1609a580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:03:58.668 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.663+0000 7fcc1609a580 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T10:03:58.668 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.663+0000 7fcc1609a580 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T10:03:58.668 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.663+0000 7fcc1609a580 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T10:03:58.737 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:03:58.732+0000 7fcc1609a580 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-01-25T10:04:04.551 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-01-25T10:04:04.551 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:04:04.552 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2023-01-25T10:04:04.553 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd3 key to auth repository 2023-01-25T10:04:04.553 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:04:04.944 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2023-01-25T10:04:04.945 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.3 2023-01-25T10:04:04.945 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:04:04.946 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:04:04.946 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:04:04.946 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:04:04.969 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:04.965+0000 7fec29419580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:04.972 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:04.967+0000 7fec29419580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:04.972 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:04.968+0000 7fec29419580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:05.273 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2023-01-25T10:04:05.274 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:04:05.274 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=3 2023-01-25T10:04:05.274 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:04:05.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:04:05.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:05.275 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:04:05.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:04:05.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:05.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:05.486 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:05.481+0000 7fec29419580 -1 Falling back to public interface 2023-01-25T10:04:05.597 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:06.598 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:06.599 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:06.599 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:04:06.599 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:04:06.600 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:06.600 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:06.910 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:07.836 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:07.831+0000 7fec29419580 -1 osd.3 0 log_to_monitors true 2023-01-25T10:04:07.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:07.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:07.912 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:04:07.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:04:07.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:07.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:08.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:09.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:09.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:09.337 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:04:09.338 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:04:09.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:09.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:09.650 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:10.651 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:10.652 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:10.652 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:04:10.652 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:04:10.653 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:10.653 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:11.077 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:12.079 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:12.079 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:12.079 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:04:12.079 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:04:12.080 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:12.080 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:12.439 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:13.440 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:13.440 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:13.440 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:04:13.441 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:04:13.441 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:13.442 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:13.852 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:14.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:14.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:14.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:04:14.855 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:04:14.855 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:14.855 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:15.177 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:16.178 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:16.178 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:16.178 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:04:16.179 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:04:16.179 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:16.179 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:16.604 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:17.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:17.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:17.606 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:04:17.606 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:04:17.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:17.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:17.918 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:18.920 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:18.920 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:18.921 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:04:18.921 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:04:18.922 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:18.922 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:19.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:20.233 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:20.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:20.234 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:04:20.234 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:04:20.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:20.235 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:20.789 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:21.790 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:21.790 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:21.791 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:04:21.791 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:04:21.791 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:21.792 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:22.342 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:23.344 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:23.344 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:23.344 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:04:23.345 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:04:23.345 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:23.346 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:23.729 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:24.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:24.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:24.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:04:24.732 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:04:24.732 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:24.733 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:25.086 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:26.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:26.087 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:26.088 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:04:26.088 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:04:26.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:26.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:26.605 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:27.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:27.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:27.607 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:04:27.608 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:04:27.608 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:27.608 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:28.155 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:29.157 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:29.157 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:29.158 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:04:29.158 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:04:29.158 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:29.159 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:29.622 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:30.623 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:30.624 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:30.624 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:04:30.625 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:04:30.625 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:30.625 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T10:04:30.959 INFO:tasks.workunit.client.0.smithi101.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/132416,v1:127.0.0.1:6827/132416] [v2:127.0.0.1:6828/132416,v1:127.0.0.1:6829/132416] exists,up bd097d5c-d859-4230-a678-81b9c511959e 2023-01-25T10:04:30.966 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:04:30.967 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:04:30.967 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:04:30.968 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:04:30.968 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:04:30.968 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:04:30.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:04:30.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-01-25T10:04:30.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:04:30.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2023-01-25T10:04:30.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:04:30.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:04:30.971 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:04:30.972 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:04:30.972 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2023-01-25T10:04:30.973 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2023-01-25T10:04:30.973 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:04:30.973 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:04:30.974 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:04:30.974 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:04:30.974 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:04:30.975 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:04:30.975 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:04:30.975 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:04:30.976 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:04:30.976 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:04:30.976 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:04:30.977 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:04:30.977 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:04:30.978 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:04:30.978 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:04:30.978 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:04:30.978 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:04:30.978 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:04:30.979 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:04:30.979 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:04:30.979 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:04:30.979 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/4 2023-01-25T10:04:30.979 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:04:30.980 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=97a151d8-c2a2-487c-b454-001fdbdeae5e 2023-01-25T10:04:30.980 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 97a151d8-c2a2-487c-b454-001fdbdeae5e' 2023-01-25T10:04:30.981 INFO:tasks.workunit.client.0.smithi101.stdout:add osd4 97a151d8-c2a2-487c-b454-001fdbdeae5e 2023-01-25T10:04:30.982 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:04:30.982 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCu/tBjrg8fOhAAT5Z7rendTeD1+8DuOjbscg== 2023-01-25T10:04:30.982 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCu/tBjrg8fOhAAT5Z7rendTeD1+8DuOjbscg=="}' 2023-01-25T10:04:30.982 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 97a151d8-c2a2-487c-b454-001fdbdeae5e -i td/osd-recovery-scrub/4/new.json 2023-01-25T10:04:31.390 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:04:31.402 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/4/new.json 2023-01-25T10:04:31.403 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQCu/tBjrg8fOhAAT5Z7rendTeD1+8DuOjbscg== --osd-uuid 97a151d8-c2a2-487c-b454-001fdbdeae5e 2023-01-25T10:04:31.429 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.424+0000 7f4117be5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:31.448 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.443+0000 7f4117be5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:31.449 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.443+0000 7f4117be5580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:31.449 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.444+0000 7f4117be5580 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-01-25T10:04:31.450 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.444+0000 7f4117be5580 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-01-25T10:04:31.450 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.444+0000 7f4117be5580 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-01-25T10:04:31.522 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:31.517+0000 7f4117be5580 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2023-01-25T10:04:37.549 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2023-01-25T10:04:37.549 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:04:37.549 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd4 key to auth repository 2023-01-25T10:04:37.550 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2023-01-25T10:04:37.550 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:04:37.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2023-01-25T10:04:37.936 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.4 2023-01-25T10:04:37.937 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:04:37.937 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:04:37.938 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:04:37.938 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:04:37.960 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:37.955+0000 7f12e95cd580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:37.961 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:37.957+0000 7f12e95cd580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:37.962 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:37.958+0000 7f12e95cd580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:04:38.274 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2023-01-25T10:04:38.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:04:38.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=4 2023-01-25T10:04:38.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:04:38.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:04:38.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:38.276 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:04:38.276 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:04:38.277 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:38.277 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:38.480 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:38.476+0000 7f12e95cd580 -1 Falling back to public interface 2023-01-25T10:04:38.582 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:39.584 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:39.584 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:39.585 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:04:39.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:04:39.586 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:39.586 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:39.891 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:40.892 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:04:40.886+0000 7f12e95cd580 -1 osd.4 0 log_to_monitors true 2023-01-25T10:04:40.892 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:40.892 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:40.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:04:40.893 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:04:40.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:40.894 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:41.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:42.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:42.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:42.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:04:42.216 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:04:42.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:42.217 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:42.588 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:43.589 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:43.590 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:43.590 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:04:43.591 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:04:43.591 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:43.591 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:43.897 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:44.898 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:44.899 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:44.899 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:04:44.899 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:04:44.900 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:44.900 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:45.288 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:46.290 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:46.290 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:46.290 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:04:46.291 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:04:46.292 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:46.292 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:46.648 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:47.649 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:47.650 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:47.650 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:04:47.650 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:04:47.651 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:47.651 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:47.962 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:48.964 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:48.964 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:48.964 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:04:48.965 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:04:48.965 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:48.966 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:49.275 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:50.277 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:50.277 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:50.278 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:04:50.278 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:04:50.279 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:50.279 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:50.592 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:51.594 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:51.594 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:51.595 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:04:51.595 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:04:51.596 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:51.596 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:51.969 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:52.970 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:52.971 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:52.971 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:04:52.971 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:04:52.972 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:52.972 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:53.871 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:54.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:54.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:54.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:04:54.874 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:04:54.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:54.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:55.208 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:56.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:56.210 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:56.211 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:04:56.211 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:04:56.212 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:56.212 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:56.558 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:57.559 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:57.559 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:57.560 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:04:57.560 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:04:57.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:57.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:57.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:04:58.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:04:58.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:04:58.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:04:58.875 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:04:58.875 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:04:58.876 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:04:59.194 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:00.195 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:00.195 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:00.196 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:05:00.196 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:05:00.197 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:00.197 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:05:00.927 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:01.929 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:01.929 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:01.929 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:05:01.930 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:05:01.930 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:01.931 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:05:02.544 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:03.545 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:03.545 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:03.546 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:05:03.546 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:05:03.547 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:03.547 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T10:05:03.902 INFO:tasks.workunit.client.0.smithi101.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/134187,v1:127.0.0.1:6835/134187] [v2:127.0.0.1:6836/134187,v1:127.0.0.1:6837/134187] exists,up 97a151d8-c2a2-487c-b454-001fdbdeae5e 2023-01-25T10:05:03.908 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:05:03.909 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:05:03.909 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:05:03.909 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:05:03.909 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:05:03.910 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:05:03.910 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:05:03.910 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2023-01-25T10:05:03.910 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:05:03.910 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2023-01-25T10:05:03.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:05:03.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:05:03.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:05:03.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:05:03.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2023-01-25T10:05:03.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2023-01-25T10:05:03.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:05:03.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:05:03.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:05:03.913 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:05:03.913 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:05:03.913 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:05:03.913 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:05:03.913 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:05:03.914 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:05:03.914 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:05:03.914 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:05:03.914 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:05:03.914 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:05:03.915 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:05:03.915 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:05:03.915 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:05:03.915 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:05:03.915 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:05:03.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:05:03.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:05:03.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:05:03.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/5 2023-01-25T10:05:03.916 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:05:03.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d20b8e35-2bf1-4f99-88e7-e9a84c54b4ee 2023-01-25T10:05:03.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd5 d20b8e35-2bf1-4f99-88e7-e9a84c54b4ee' 2023-01-25T10:05:03.917 INFO:tasks.workunit.client.0.smithi101.stdout:add osd5 d20b8e35-2bf1-4f99-88e7-e9a84c54b4ee 2023-01-25T10:05:03.919 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:05:03.922 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDP/tBjgaO5NhAAapqHe+KxwhHzyjc64oBWWQ== 2023-01-25T10:05:03.922 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDP/tBjgaO5NhAAapqHe+KxwhHzyjc64oBWWQ=="}' 2023-01-25T10:05:03.923 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d20b8e35-2bf1-4f99-88e7-e9a84c54b4ee -i td/osd-recovery-scrub/5/new.json 2023-01-25T10:05:04.322 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:05:04.334 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/5/new.json 2023-01-25T10:05:04.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 5 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQDP/tBjgaO5NhAAapqHe+KxwhHzyjc64oBWWQ== --osd-uuid d20b8e35-2bf1-4f99-88e7-e9a84c54b4ee 2023-01-25T10:05:04.362 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.358+0000 7faaf59ef580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:04.380 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.376+0000 7faaf59ef580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:04.381 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.377+0000 7faaf59ef580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:04.382 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.378+0000 7faaf59ef580 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-01-25T10:05:04.382 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.378+0000 7faaf59ef580 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-01-25T10:05:04.382 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.378+0000 7faaf59ef580 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-01-25T10:05:04.451 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:04.447+0000 7faaf59ef580 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2023-01-25T10:05:10.538 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2023-01-25T10:05:10.539 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:05:10.539 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd5 key to auth repository 2023-01-25T10:05:10.540 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd5 key to auth repository 2023-01-25T10:05:10.540 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:05:10.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.5 2023-01-25T10:05:10.936 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.5 2023-01-25T10:05:10.937 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 5 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:05:10.937 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:05:10.938 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:05:10.938 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:05:10.958 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:10.953+0000 7fa7aa7d7580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:10.960 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:10.955+0000 7fa7aa7d7580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:10.960 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:10.956+0000 7fa7aa7d7580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:11.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 5 2023-01-25T10:05:11.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:05:11.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=5 2023-01-25T10:05:11.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:05:11.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:05:11.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:11.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:05:11.267 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:05:11.272 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:11.272 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:11.480 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:11.475+0000 7fa7aa7d7580 -1 Falling back to public interface 2023-01-25T10:05:11.570 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:12.571 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:12.571 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:12.572 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:05:12.572 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:05:12.573 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:12.573 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:12.880 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:13.786 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:13.781+0000 7fa7aa7d7580 -1 osd.5 0 log_to_monitors true 2023-01-25T10:05:13.881 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:13.882 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:13.882 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:05:13.882 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:05:13.883 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:13.883 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:14.260 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:15.263 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:15.263 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:15.264 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:05:15.265 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:05:15.265 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:15.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:15.576 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:16.577 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:16.577 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:16.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:05:16.578 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:05:16.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:16.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:16.892 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:17.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:17.894 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:17.894 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:05:17.894 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:05:17.895 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:17.895 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:18.251 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:19.252 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:05:19.253 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:19.253 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:19.253 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:05:19.253 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:19.253 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:19.567 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:20.569 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:20.569 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:20.569 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:05:20.570 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:05:20.570 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:20.570 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:20.884 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:21.885 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:21.885 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:21.886 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:05:21.886 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:05:21.887 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:21.887 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:22.242 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:23.244 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:23.365 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:23.365 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:05:23.366 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:05:23.366 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:23.366 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:23.596 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:24.598 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:24.598 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:24.599 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:05:24.599 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:05:24.599 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:24.600 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:24.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:25.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:25.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:25.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:05:25.917 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:05:25.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:25.918 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:26.495 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:27.496 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:27.496 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:27.497 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:05:27.497 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:05:27.497 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:27.498 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:28.035 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:29.039 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:29.040 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:29.040 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:05:29.041 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:05:29.041 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:29.041 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:29.454 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:30.456 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:30.456 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:30.456 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:05:30.457 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:05:30.457 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:30.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:31.089 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:32.091 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:32.091 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:32.091 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:05:32.092 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:05:32.092 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:32.092 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:32.449 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:33.450 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:33.451 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:33.451 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:05:33.451 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:05:33.452 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:33.452 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:33.911 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:34.912 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:34.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:34.913 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:05:34.913 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:05:34.914 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:34.914 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:35.336 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:36.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:36.338 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:36.339 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:05:36.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:05:36.339 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:36.340 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T10:05:36.650 INFO:tasks.workunit.client.0.smithi101.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/135958,v1:127.0.0.1:6843/135958] [v2:127.0.0.1:6844/135958,v1:127.0.0.1:6845/135958] exists,up d20b8e35-2bf1-4f99-88e7-e9a84c54b4ee 2023-01-25T10:05:36.654 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:05:36.654 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:05:36.655 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:05:36.655 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:05:36.655 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:05:36.655 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:05:36.655 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:05:36.656 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2023-01-25T10:05:36.656 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:05:36.656 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2023-01-25T10:05:36.656 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:05:36.657 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:05:36.657 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:05:36.657 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:05:36.657 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2023-01-25T10:05:36.658 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2023-01-25T10:05:36.658 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:05:36.658 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:05:36.659 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:05:36.659 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:05:36.659 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:05:36.659 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:05:36.660 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:05:36.660 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:05:36.660 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:05:36.660 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:05:36.660 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:05:36.661 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:05:36.661 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:05:36.661 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:05:36.661 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:05:36.661 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:05:36.662 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:05:36.662 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:05:36.662 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:05:36.662 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:05:36.662 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:05:36.663 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/6 2023-01-25T10:05:36.663 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:05:36.663 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f48b51fb-a552-492c-9778-35231ab8eab1 2023-01-25T10:05:36.663 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd6 f48b51fb-a552-492c-9778-35231ab8eab1' 2023-01-25T10:05:36.664 INFO:tasks.workunit.client.0.smithi101.stdout:add osd6 f48b51fb-a552-492c-9778-35231ab8eab1 2023-01-25T10:05:36.664 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:05:36.670 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDw/tBjEC6vJxAAjqIVCv75ElTByRK/2kYGQA== 2023-01-25T10:05:36.670 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDw/tBjEC6vJxAAjqIVCv75ElTByRK/2kYGQA=="}' 2023-01-25T10:05:36.670 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f48b51fb-a552-492c-9778-35231ab8eab1 -i td/osd-recovery-scrub/6/new.json 2023-01-25T10:05:37.068 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:05:37.081 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/6/new.json 2023-01-25T10:05:37.082 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 6 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQDw/tBjEC6vJxAAjqIVCv75ElTByRK/2kYGQA== --osd-uuid f48b51fb-a552-492c-9778-35231ab8eab1 2023-01-25T10:05:37.109 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.105+0000 7f0d827fa580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:37.122 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.118+0000 7f0d827fa580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:37.123 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.119+0000 7f0d827fa580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:37.123 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.119+0000 7f0d827fa580 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-01-25T10:05:37.124 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.119+0000 7f0d827fa580 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-01-25T10:05:37.124 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.119+0000 7f0d827fa580 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-01-25T10:05:37.243 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:37.239+0000 7f0d827fa580 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2023-01-25T10:05:43.492 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2023-01-25T10:05:43.492 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:05:43.493 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd6 key to auth repository 2023-01-25T10:05:43.494 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd6 key to auth repository 2023-01-25T10:05:43.494 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:05:43.883 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.6 2023-01-25T10:05:43.884 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.6 2023-01-25T10:05:43.885 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 6 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:05:43.885 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:05:43.885 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:05:43.886 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:05:43.908 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:43.904+0000 7fc79d2c4580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:43.911 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:43.907+0000 7fc79d2c4580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:43.912 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:43.908+0000 7fc79d2c4580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:05:44.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 6 2023-01-25T10:05:44.226 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:05:44.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=6 2023-01-25T10:05:44.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:05:44.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:05:44.227 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:44.228 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:05:44.228 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:05:44.228 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:44.229 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:44.427 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:44.423+0000 7fc79d2c4580 -1 Falling back to public interface 2023-01-25T10:05:44.558 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:45.560 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:45.560 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:45.560 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:05:45.561 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:05:45.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:45.562 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:45.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:46.781 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:05:46.777+0000 7fc79d2c4580 -1 osd.6 0 log_to_monitors true 2023-01-25T10:05:46.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:46.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:46.874 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:05:46.875 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:05:46.875 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:46.875 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:47.264 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:48.265 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:48.265 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:48.266 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:05:48.266 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:05:48.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:48.267 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:48.582 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:49.584 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:49.584 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:49.584 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:05:49.585 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:05:49.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:49.585 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:49.902 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:50.903 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:50.903 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:50.904 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:05:50.904 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:05:50.905 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:50.905 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:51.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:52.217 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:52.217 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:52.217 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:05:52.218 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:05:52.218 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:52.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:52.569 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:53.570 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:53.571 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:53.571 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:05:53.571 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:05:53.572 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:53.572 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:53.884 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:54.886 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:54.886 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:54.886 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:05:54.887 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:05:54.887 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:54.887 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:55.198 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:56.199 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:56.199 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:56.200 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:05:56.200 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:05:56.200 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:56.201 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:56.620 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:57.621 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:57.621 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:57.622 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:05:57.622 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:05:57.622 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:57.623 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:57.933 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:05:58.934 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:05:58.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:05:58.935 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:05:58.935 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:05:58.936 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:05:58.936 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:05:59.458 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:00.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:00.460 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:00.461 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:06:00.461 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:06:00.461 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:00.462 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:00.794 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:01.796 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:01.796 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:01.797 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:06:01.797 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:06:01.798 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:01.798 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:02.641 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:03.643 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:03.643 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:03.643 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:06:03.644 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:06:03.644 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:03.644 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:04.109 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:05.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:05.110 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:05.111 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:06:05.111 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:06:05.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:05.112 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:05.535 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:06.537 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:06.537 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:06.537 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:06:06.538 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:06:06.538 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:06.539 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:06.871 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:07.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:07.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:07.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:06:07.873 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:06:07.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:07.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:08.483 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:09.484 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:09.484 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:09.484 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:06:09.485 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:06:09.485 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:09.486 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T10:06:09.832 INFO:tasks.workunit.client.0.smithi101.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/137721,v1:127.0.0.1:6851/137721] [v2:127.0.0.1:6852/137721,v1:127.0.0.1:6853/137721] exists,up f48b51fb-a552-492c-9778-35231ab8eab1 2023-01-25T10:06:09.836 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:06:09.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:06:09.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:06:09.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T10:06:09.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:06:09.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T10:06:09.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T10:06:09.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2023-01-25T10:06:09.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T10:06:09.838 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2023-01-25T10:06:09.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T10:06:09.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T10:06:09.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T10:06:09.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T10:06:09.839 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2023-01-25T10:06:09.840 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2023-01-25T10:06:09.840 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T10:06:09.840 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T10:06:09.840 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T10:06:09.840 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T10:06:09.840 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T10:06:09.841 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T10:06:09.841 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T10:06:09.841 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T10:06:09.841 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.115908 2023-01-25T10:06:09.841 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:06:09.842 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' 2023-01-25T10:06:09.842 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T10:06:09.842 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T10:06:09.842 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T10:06:09.842 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T10:06:09.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T10:06:09.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T10:06:09.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T10:06:09.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T10:06:09.843 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T10:06:09.844 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T10:06:09.844 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/7 2023-01-25T10:06:09.844 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T10:06:09.844 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=053cdfe8-579b-46b1-86b1-48bf13889bda 2023-01-25T10:06:09.844 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd7 053cdfe8-579b-46b1-86b1-48bf13889bda' 2023-01-25T10:06:09.845 INFO:tasks.workunit.client.0.smithi101.stdout:add osd7 053cdfe8-579b-46b1-86b1-48bf13889bda 2023-01-25T10:06:09.845 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T10:06:09.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAR/9BjKmKaMhAAgIeKyi4y7jhfHd0tLyjJ0w== 2023-01-25T10:06:09.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAR/9BjKmKaMhAAgIeKyi4y7jhfHd0tLyjJ0w=="}' 2023-01-25T10:06:09.854 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 053cdfe8-579b-46b1-86b1-48bf13889bda -i td/osd-recovery-scrub/7/new.json 2023-01-25T10:06:10.232 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:06:10.244 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/7/new.json 2023-01-25T10:06:10.245 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 7 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAR/9BjKmKaMhAAgIeKyi4y7jhfHd0tLyjJ0w== --osd-uuid 053cdfe8-579b-46b1-86b1-48bf13889bda 2023-01-25T10:06:10.271 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.266+0000 7f8ddd44b580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:06:10.290 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.285+0000 7f8ddd44b580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:06:10.290 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.286+0000 7f8ddd44b580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:06:10.291 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.286+0000 7f8ddd44b580 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-01-25T10:06:10.291 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.286+0000 7f8ddd44b580 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-01-25T10:06:10.291 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.286+0000 7f8ddd44b580 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-01-25T10:06:10.369 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:10.364+0000 7f8ddd44b580 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2023-01-25T10:06:16.474 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2023-01-25T10:06:16.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T10:06:16.475 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd7 key to auth repository 2023-01-25T10:06:16.476 INFO:tasks.workunit.client.0.smithi101.stdout:adding osd7 key to auth repository 2023-01-25T10:06:16.476 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T10:06:16.870 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.7 2023-01-25T10:06:16.871 INFO:tasks.workunit.client.0.smithi101.stdout:start osd.7 2023-01-25T10:06:16.871 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 7 --fsid=1f72c063-b43f-4dc2-b5e2-137fa815847f --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.115908/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T10:06:16.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T10:06:16.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T10:06:16.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T10:06:16.894 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:16.890+0000 7f21f3139580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:06:16.896 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:16.893+0000 7f21f3139580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:06:16.897 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:16.893+0000 7f21f3139580 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T10:06:17.213 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 7 2023-01-25T10:06:17.213 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T10:06:17.213 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=7 2023-01-25T10:06:17.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T10:06:17.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T10:06:17.214 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:17.215 INFO:tasks.workunit.client.0.smithi101.stdout:0 2023-01-25T10:06:17.215 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T10:06:17.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:17.216 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:17.411 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:17.406+0000 7f21f3139580 -1 Falling back to public interface 2023-01-25T10:06:17.515 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:18.517 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:18.517 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:18.517 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T10:06:18.517 INFO:tasks.workunit.client.0.smithi101.stdout:1 2023-01-25T10:06:18.518 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:18.518 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:18.834 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:19.771 INFO:tasks.workunit.client.0.smithi101.stderr:2023-01-25T10:06:19.767+0000 7f21f3139580 -1 osd.7 0 log_to_monitors true 2023-01-25T10:06:19.835 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:19.835 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:19.836 INFO:tasks.workunit.client.0.smithi101.stdout:2 2023-01-25T10:06:19.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T10:06:19.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:19.837 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:20.240 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:21.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:21.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:21.239 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T10:06:21.240 INFO:tasks.workunit.client.0.smithi101.stdout:3 2023-01-25T10:06:21.240 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:21.241 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:21.555 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:22.556 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:22.557 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:22.557 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T10:06:22.557 INFO:tasks.workunit.client.0.smithi101.stdout:4 2023-01-25T10:06:22.558 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:22.558 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:22.870 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:23.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:23.872 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:23.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T10:06:23.873 INFO:tasks.workunit.client.0.smithi101.stdout:5 2023-01-25T10:06:23.873 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:23.874 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:24.247 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:25.248 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:25.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:25.249 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T10:06:25.249 INFO:tasks.workunit.client.0.smithi101.stdout:6 2023-01-25T10:06:25.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:25.250 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:25.576 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:26.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:26.578 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:26.578 INFO:tasks.workunit.client.0.smithi101.stdout:7 2023-01-25T10:06:26.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T10:06:26.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:26.579 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:26.891 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:27.892 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:27.892 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:27.893 INFO:tasks.workunit.client.0.smithi101.stdout:8 2023-01-25T10:06:27.893 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T10:06:27.894 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:27.894 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:28.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:29.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:29.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:29.221 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T10:06:29.221 INFO:tasks.workunit.client.0.smithi101.stdout:9 2023-01-25T10:06:29.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:29.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:29.572 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:30.573 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:30.574 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:30.574 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T10:06:30.574 INFO:tasks.workunit.client.0.smithi101.stdout:10 2023-01-25T10:06:30.575 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:30.575 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:30.886 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:31.888 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:31.888 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:31.888 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T10:06:31.889 INFO:tasks.workunit.client.0.smithi101.stdout:11 2023-01-25T10:06:31.890 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:31.890 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:32.507 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:33.509 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:33.509 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:33.509 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T10:06:33.510 INFO:tasks.workunit.client.0.smithi101.stdout:12 2023-01-25T10:06:33.510 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:33.510 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:33.823 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:34.825 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:34.825 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:34.825 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T10:06:34.826 INFO:tasks.workunit.client.0.smithi101.stdout:13 2023-01-25T10:06:34.826 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:34.826 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:35.222 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:36.223 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:36.223 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:36.224 INFO:tasks.workunit.client.0.smithi101.stdout:14 2023-01-25T10:06:36.224 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T10:06:36.224 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:36.225 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:36.810 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:37.811 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:37.812 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:37.812 INFO:tasks.workunit.client.0.smithi101.stdout:15 2023-01-25T10:06:37.813 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T10:06:37.813 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:37.813 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:38.137 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:39.138 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:39.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:39.139 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T10:06:39.139 INFO:tasks.workunit.client.0.smithi101.stdout:16 2023-01-25T10:06:39.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:39.140 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:39.731 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:40.733 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:40.733 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:40.733 INFO:tasks.workunit.client.0.smithi101.stdout:17 2023-01-25T10:06:40.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T10:06:40.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:40.734 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:41.150 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T10:06:42.151 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T10:06:42.152 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T10:06:42.152 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T10:06:42.152 INFO:tasks.workunit.client.0.smithi101.stdout:18 2023-01-25T10:06:42.153 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T10:06:42.153 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T10:06:42.515 INFO:tasks.workunit.client.0.smithi101.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/139492,v1:127.0.0.1:6859/139492] [v2:127.0.0.1:6860/139492,v1:127.0.0.1:6861/139492] exists,up 053cdfe8-579b-46b1-86b1-48bf13889bda 2023-01-25T10:06:42.516 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T10:06:42.516 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T10:06:42.516 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T10:06:42.517 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: create_pool test 32 32 2023-01-25T10:06:42.517 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2023-01-25T10:06:43.117 INFO:tasks.workunit.client.0.smithi101.stderr:pool 'test' created 2023-01-25T10:06:43.127 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T10:06:44.129 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: wait_for_clean 2023-01-25T10:06:44.129 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T10:06:44.129 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T10:06:44.129 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T10:06:44.130 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T10:06:44.130 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T10:06:44.130 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T10:06:44.130 INFO:tasks.workunit.client.0.smithi101.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T10:06:44.130 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T10:06:44.131 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T10:06:44.131 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T10:06:44.219 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T10:06:44.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T10:06:44.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T10:06:44.220 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T10:06:44.220 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T10:06:44.558 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T10:06:44.559 INFO:tasks.workunit.client.0.smithi101.stderr:1 2023-01-25T10:06:44.559 INFO:tasks.workunit.client.0.smithi101.stderr:2 2023-01-25T10:06:44.560 INFO:tasks.workunit.client.0.smithi101.stderr:3 2023-01-25T10:06:44.560 INFO:tasks.workunit.client.0.smithi101.stderr:4 2023-01-25T10:06:44.560 INFO:tasks.workunit.client.0.smithi101.stderr:5 2023-01-25T10:06:44.561 INFO:tasks.workunit.client.0.smithi101.stderr:6 2023-01-25T10:06:44.561 INFO:tasks.workunit.client.0.smithi101.stderr:7' 2023-01-25T10:06:44.561 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T10:06:44.562 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:44.562 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T10:06:44.682 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836533 2023-01-25T10:06:44.682 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836533 2023-01-25T10:06:44.683 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533' 2023-01-25T10:06:44.683 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:44.683 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T10:06:45.100 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673009 2023-01-25T10:06:45.100 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673009 2023-01-25T10:06:45.100 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009' 2023-01-25T10:06:45.101 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:45.101 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T10:06:45.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509476 2023-01-25T10:06:45.335 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509476 2023-01-25T10:06:45.336 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509476' 2023-01-25T10:06:45.336 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:45.336 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T10:06:45.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345951 2023-01-25T10:06:45.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345951 2023-01-25T10:06:45.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509476 3-85899345951' 2023-01-25T10:06:45.467 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:45.468 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-25T10:06:45.668 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182428 2023-01-25T10:06:45.669 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182428 2023-01-25T10:06:45.669 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509476 3-85899345951 4-107374182428' 2023-01-25T10:06:45.669 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:45.669 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2023-01-25T10:06:45.793 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=128849018903 2023-01-25T10:06:45.794 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 128849018903 2023-01-25T10:06:45.794 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509476 3-85899345951 4-107374182428 5-128849018903' 2023-01-25T10:06:45.794 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:45.794 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2023-01-25T10:06:45.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=150323855373 2023-01-25T10:06:45.916 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 150323855373 2023-01-25T10:06:45.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509476 3-85899345951 4-107374182428 5-128849018903 6-150323855373' 2023-01-25T10:06:45.917 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T10:06:45.917 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2023-01-25T10:06:46.039 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=171798691849 2023-01-25T10:06:46.040 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 171798691849 2023-01-25T10:06:46.040 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509476 3-85899345951 4-107374182428 5-128849018903 6-150323855373 7-171798691849' 2023-01-25T10:06:46.040 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:46.040 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836533 2023-01-25T10:06:46.041 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:46.041 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T10:06:46.041 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836533 2023-01-25T10:06:46.042 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:46.042 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836533 2023-01-25T10:06:46.042 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2023-01-25T10:06:46.043 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.0 seq 21474836533 2023-01-25T10:06:46.043 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T10:06:46.345 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836533 -lt 21474836533 2023-01-25T10:06:46.345 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:46.345 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673009 2023-01-25T10:06:46.346 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:46.346 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T10:06:46.346 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673009 2023-01-25T10:06:46.347 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:46.347 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673009 2023-01-25T10:06:46.348 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2023-01-25T10:06:46.348 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.1 seq 42949673009 2023-01-25T10:06:46.348 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T10:06:46.676 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673009 -lt 42949673009 2023-01-25T10:06:46.676 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:46.677 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509476 2023-01-25T10:06:46.677 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:46.678 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T10:06:46.678 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509476 2023-01-25T10:06:46.678 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:46.679 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509476 2023-01-25T10:06:46.679 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2023-01-25T10:06:46.679 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.2 seq 64424509476 2023-01-25T10:06:46.680 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T10:06:47.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509476 -lt 64424509476 2023-01-25T10:06:47.020 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:47.021 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345951 2023-01-25T10:06:47.021 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:47.022 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T10:06:47.022 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345951 2023-01-25T10:06:47.022 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:47.023 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345951 2023-01-25T10:06:47.023 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2023-01-25T10:06:47.023 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.3 seq 85899345951 2023-01-25T10:06:47.024 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T10:06:47.361 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345951 -lt 85899345951 2023-01-25T10:06:47.362 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:47.362 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182428 2023-01-25T10:06:47.362 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:47.363 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-25T10:06:47.363 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182428 2023-01-25T10:06:47.363 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:47.364 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182428 2023-01-25T10:06:47.364 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2023-01-25T10:06:47.365 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.4 seq 107374182428 2023-01-25T10:06:47.365 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-25T10:06:47.709 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182428 -lt 107374182428 2023-01-25T10:06:47.709 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:47.710 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 5-128849018903 2023-01-25T10:06:47.710 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:47.711 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=5 2023-01-25T10:06:47.711 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 5-128849018903 2023-01-25T10:06:47.711 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:47.712 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=128849018903 2023-01-25T10:06:47.712 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.5 seq 128849018903' 2023-01-25T10:06:47.712 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.5 seq 128849018903 2023-01-25T10:06:47.713 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 5 2023-01-25T10:06:48.065 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 128849018903 -lt 128849018903 2023-01-25T10:06:48.065 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:48.066 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 6-150323855373 2023-01-25T10:06:48.066 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:48.067 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=6 2023-01-25T10:06:48.068 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 6-150323855373 2023-01-25T10:06:48.068 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:48.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=150323855373 2023-01-25T10:06:48.068 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.6 seq 150323855373' 2023-01-25T10:06:48.069 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.6 seq 150323855373 2023-01-25T10:06:48.070 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 6 2023-01-25T10:06:48.389 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 150323855373 -lt 150323855373 2023-01-25T10:06:48.390 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T10:06:48.390 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 7-171798691849 2023-01-25T10:06:48.390 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T10:06:48.391 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=7 2023-01-25T10:06:48.391 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 7-171798691849 2023-01-25T10:06:48.391 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T10:06:48.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=171798691849 2023-01-25T10:06:48.392 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.7 seq 171798691849' 2023-01-25T10:06:48.392 INFO:tasks.workunit.client.0.smithi101.stdout:waiting osd.7 seq 171798691849 2023-01-25T10:06:48.393 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 7 2023-01-25T10:06:48.694 INFO:tasks.workunit.client.0.smithi101.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 171798691849 -lt 171798691849 2023-01-25T10:06:48.694 INFO:tasks.workunit.client.0.smithi101.stderr://home/ubuntu/cephtest/clon