2023-01-25T02:48:46.814 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-25T02:48:46.815 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} --verbose --archive /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136616 --name yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136616/orig.config.yaml 2023-01-25T02:48:46.838 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T02:48:46.891 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136616 branch: wip-yuri2-testing-2023-01-23-0928 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: yweinstein@radhat.com first_in_suite: false job_id: '7136616' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri2-testing-2023-01-23-0928 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 61996d73ab738798e4ee0055c86457f5c459da4a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 61996d73ab738798e4ee0055c86457f5c459da4a workunit: branch: wip-yuri2-testing-2023-01-23-0928 sha1: 61996d73ab738798e4ee0055c86457f5c459da4a owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 61996d73ab738798e4ee0055c86457f5c459da4a sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2023-01-23-0928 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 61996d73ab738798e4ee0055c86457f5c459da4a targets: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_22:22:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-25T02:48:46.891 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa; will attempt to use it 2023-01-25T02:48:46.892 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa/tasks 2023-01-25T02:48:46.892 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-25T02:48:46.894 INFO:teuthology.task.internal:Checking packages... 2023-01-25T02:48:46.920 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '61996d73ab738798e4ee0055c86457f5c459da4a' 2023-01-25T02:48:46.920 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-25T02:48:46.921 INFO:teuthology.packaging:ref: None 2023-01-25T02:48:46.921 INFO:teuthology.packaging:tag: None 2023-01-25T02:48:46.921 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-01-23-0928 2023-01-25T02:48:46.921 INFO:teuthology.packaging:sha1: 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T02:48:46.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2023-01-23-0928 2023-01-25T02:48:47.071 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1963-g61996d73-1focal 2023-01-25T02:48:47.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-25T02:48:47.079 INFO:teuthology.task.internal:no buildpackages task found 2023-01-25T02:48:47.080 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-25T02:48:47.100 INFO:teuthology.task.internal:Saving configuration 2023-01-25T02:48:47.110 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-25T02:48:47.125 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T02:48:47.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136616', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 02:42:48.624463', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-01-25T02:48:47.149 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-25T02:48:47.156 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-25T02:48:47.156 INFO:teuthology.run_tasks:Running task console_log... 2023-01-25T02:48:47.212 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f40cc27a5e0>, signals=[15]) 2023-01-25T02:48:47.213 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-25T02:48:47.219 INFO:teuthology.task.internal:Opening connections... 2023-01-25T02:48:47.219 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2023-01-25T02:48:47.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T02:48:47.279 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-25T02:48:47.286 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2023-01-25T02:48:47.398 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2023-01-25T02:48:47.399 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2023-01-25T02:48:47.405 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Ubuntu" 2023-01-25T02:48:47.405 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-25T02:48:47.405 INFO:teuthology.orchestra.run.smithi008.stdout:ID=ubuntu 2023-01-25T02:48:47.405 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE=debian 2023-01-25T02:48:47.406 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-25T02:48:47.406 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="20.04" 2023-01-25T02:48:47.406 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-25T02:48:47.406 INFO:teuthology.orchestra.run.smithi008.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-25T02:48:47.406 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-25T02:48:47.407 INFO:teuthology.orchestra.run.smithi008.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-25T02:48:47.407 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_CODENAME=focal 2023-01-25T02:48:47.407 INFO:teuthology.orchestra.run.smithi008.stdout:UBUNTU_CODENAME=focal 2023-01-25T02:48:47.408 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2023-01-25T02:48:47.428 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-25T02:48:47.437 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-25T02:48:47.458 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-25T02:48:47.459 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2023-01-25T02:48:47.465 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-25T02:48:47.483 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-25T02:48:47.484 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2023-01-25T02:48:47.500 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-25T02:48:47.548 INFO:teuthology.run_tasks:Running task kernel... 2023-01-25T02:48:47.563 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-25T02:48:47.564 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-25T02:48:47.564 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T02:48:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T02:48:47.571 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2023-01-25T02:48:47.621 INFO:teuthology.orchestra.run.smithi008.stdout:5.4.0-124-generic 2023-01-25T02:48:47.621 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.4.0-124-generic 2023-01-25T02:48:47.621 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get clean 2023-01-25T02:48:47.719 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get update 2023-01-25T02:48:48.036 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-25T02:48:48.039 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-01-25T02:48:48.095 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-01-25T02:48:48.481 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-01-25T02:48:48.553 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [543 kB] 2023-01-25T02:48:48.695 INFO:teuthology.orchestra.run.smithi008.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [780 kB] 2023-01-25T02:48:48.919 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,960 kB] 2023-01-25T02:48:49.047 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,341 kB] 2023-01-25T02:48:49.122 INFO:teuthology.orchestra.run.smithi008.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [320 kB] 2023-01-25T02:48:49.137 INFO:teuthology.orchestra.run.smithi008.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.7 kB] 2023-01-25T02:48:49.138 INFO:teuthology.orchestra.run.smithi008.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-01-25T02:48:49.139 INFO:teuthology.orchestra.run.smithi008.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,463 kB] 2023-01-25T02:48:49.206 INFO:teuthology.orchestra.run.smithi008.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [207 kB] 2023-01-25T02:48:49.211 INFO:teuthology.orchestra.run.smithi008.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [624 B] 2023-01-25T02:48:49.211 INFO:teuthology.orchestra.run.smithi008.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [576 kB] 2023-01-25T02:48:49.227 INFO:teuthology.orchestra.run.smithi008.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [787 kB] 2023-01-25T02:48:49.236 INFO:teuthology.orchestra.run.smithi008.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [404 kB] 2023-01-25T02:48:49.245 INFO:teuthology.orchestra.run.smithi008.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [152 kB] 2023-01-25T02:48:49.249 INFO:teuthology.orchestra.run.smithi008.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.9 kB] 2023-01-25T02:48:49.250 INFO:teuthology.orchestra.run.smithi008.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-01-25T02:48:49.251 INFO:teuthology.orchestra.run.smithi008.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.4 kB] 2023-01-25T02:48:49.252 INFO:teuthology.orchestra.run.smithi008.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,560 kB] 2023-01-25T02:48:49.308 INFO:teuthology.orchestra.run.smithi008.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [220 kB] 2023-01-25T02:48:49.315 INFO:teuthology.orchestra.run.smithi008.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [620 B] 2023-01-25T02:48:49.316 INFO:teuthology.orchestra.run.smithi008.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [708 kB] 2023-01-25T02:48:49.334 INFO:teuthology.orchestra.run.smithi008.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,019 kB] 2023-01-25T02:48:49.362 INFO:teuthology.orchestra.run.smithi008.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [236 kB] 2023-01-25T02:48:49.367 INFO:teuthology.orchestra.run.smithi008.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.4 kB] 2023-01-25T02:48:49.409 INFO:teuthology.orchestra.run.smithi008.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.2 kB] 2023-01-25T02:48:49.409 INFO:teuthology.orchestra.run.smithi008.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-01-25T02:48:49.410 INFO:teuthology.orchestra.run.smithi008.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [604 B] 2023-01-25T02:48:49.410 INFO:teuthology.orchestra.run.smithi008.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB] 2023-01-25T02:48:49.411 INFO:teuthology.orchestra.run.smithi008.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-01-25T02:48:49.423 INFO:teuthology.orchestra.run.smithi008.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-01-25T02:48:52.522 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 13.8 MB in 3s (4,969 kB/s) 2023-01-25T02:48:53.957 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2023-01-25T02:48:53.982 DEBUG:teuthology.orchestra.run.smithi008:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-25T02:48:54.113 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2023-01-25T02:48:54.308 INFO:teuthology.orchestra.run.smithi008.stdout:Building dependency tree... 2023-01-25T02:48:54.309 INFO:teuthology.orchestra.run.smithi008.stdout:Reading state information... 2023-01-25T02:48:54.485 INFO:teuthology.orchestra.run.smithi008.stdout:The following additional packages will be installed: 2023-01-25T02:48:54.485 INFO:teuthology.orchestra.run.smithi008.stdout: linux-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-25T02:48:54.485 INFO:teuthology.orchestra.run.smithi008.stdout: linux-headers-generic linux-image-5.4.0-137-generic 2023-01-25T02:48:54.486 INFO:teuthology.orchestra.run.smithi008.stdout: linux-modules-5.4.0-137-generic linux-modules-extra-5.4.0-137-generic 2023-01-25T02:48:54.486 INFO:teuthology.orchestra.run.smithi008.stdout:Suggested packages: 2023-01-25T02:48:54.486 INFO:teuthology.orchestra.run.smithi008.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-01-25T02:48:54.486 INFO:teuthology.orchestra.run.smithi008.stdout:Recommended packages: 2023-01-25T02:48:54.487 INFO:teuthology.orchestra.run.smithi008.stdout: thermald 2023-01-25T02:48:54.546 INFO:teuthology.orchestra.run.smithi008.stdout:The following NEW packages will be installed: 2023-01-25T02:48:54.546 INFO:teuthology.orchestra.run.smithi008.stdout: linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-25T02:48:54.546 INFO:teuthology.orchestra.run.smithi008.stdout: linux-image-5.4.0-137-generic linux-modules-5.4.0-137-generic 2023-01-25T02:48:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: linux-modules-extra-5.4.0-137-generic 2023-01-25T02:48:54.547 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages will be upgraded: 2023-01-25T02:48:54.548 INFO:teuthology.orchestra.run.smithi008.stdout: linux-generic linux-headers-generic linux-image-generic 2023-01-25T02:48:54.599 INFO:teuthology.orchestra.run.smithi008.stdout:3 upgraded, 5 newly installed, 0 to remove and 129 not upgraded. 2023-01-25T02:48:54.599 INFO:teuthology.orchestra.run.smithi008.stdout:Need to get 77.1 MB of archives. 2023-01-25T02:48:54.599 INFO:teuthology.orchestra.run.smithi008.stdout:After this operation, 380 MB of additional disk space will be used. 2023-01-25T02:48:54.599 INFO:teuthology.orchestra.run.smithi008.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-137-generic amd64 5.4.0-137.154 [15.0 MB] 2023-01-25T02:48:54.924 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-137-generic amd64 5.4.0-137.154 [10.5 MB] 2023-01-25T02:48:55.019 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-137-generic amd64 5.4.0-137.154 [39.2 MB] 2023-01-25T02:48:55.498 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.137.135 [1,896 B] 2023-01-25T02:48:55.498 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.137.135 [2,528 B] 2023-01-25T02:48:55.499 INFO:teuthology.orchestra.run.smithi008.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137 all 5.4.0-137.154 [11.0 MB] 2023-01-25T02:48:55.595 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137-generic amd64 5.4.0-137.154 [1,400 kB] 2023-01-25T02:48:55.606 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.137.135 [2,392 B] 2023-01-25T02:48:56.289 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 77.1 MB in 1s (73.3 MB/s) 2023-01-25T02:48:56.498 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-modules-5.4.0-137-generic. 2023-01-25T02:48:58.204 INFO:teuthology.orchestra.run.smithi008.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-01-25T02:48:58.209 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../0-linux-modules-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T02:48:58.312 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:00.421 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-image-5.4.0-137-generic. 2023-01-25T02:49:00.436 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../1-linux-image-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T02:49:00.516 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:01.632 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-137-generic. 2023-01-25T02:49:01.649 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T02:49:01.691 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:06.548 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../3-linux-generic_5.4.0.137.135_amd64.deb ... 2023-01-25T02:49:06.789 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-25T02:49:07.299 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.137.135_amd64.deb ... 2023-01-25T02:49:07.443 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-image-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-25T02:49:07.779 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-headers-5.4.0-137. 2023-01-25T02:49:07.795 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../5-linux-headers-5.4.0-137_5.4.0-137.154_all.deb ... 2023-01-25T02:49:07.856 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-25T02:49:13.380 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package linux-headers-5.4.0-137-generic. 2023-01-25T02:49:13.401 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../6-linux-headers-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T02:49:13.438 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:16.220 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.137.135_amd64.deb ... 2023-01-25T02:49:16.349 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking linux-headers-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-25T02:49:16.713 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-25T02:49:16.837 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:18.140 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:18.291 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:19.484 INFO:teuthology.orchestra.run.smithi008.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-137-generic 2023-01-25T02:49:19.484 INFO:teuthology.orchestra.run.smithi008.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-137-generic 2023-01-25T02:49:19.618 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:20.836 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-image-generic (5.4.0.137.135) ... 2023-01-25T02:49:20.962 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-headers-generic (5.4.0.137.135) ... 2023-01-25T02:49:21.109 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up linux-generic (5.4.0.137.135) ... 2023-01-25T02:49:21.231 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T02:49:21.299 INFO:teuthology.orchestra.run.smithi008.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-01-25T02:49:21.300 INFO:teuthology.orchestra.run.smithi008.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-137-generic 2023-01-25T02:49:44.153 INFO:teuthology.orchestra.run.smithi008.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-01-25T02:49:44.153 INFO:teuthology.orchestra.run.smithi008.stdout:Sourcing file `/etc/default/grub' 2023-01-25T02:49:44.166 INFO:teuthology.orchestra.run.smithi008.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-25T02:49:44.195 INFO:teuthology.orchestra.run.smithi008.stdout:Generating grub configuration file ... 2023-01-25T02:49:44.508 INFO:teuthology.orchestra.run.smithi008.stdout:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-25T02:49:44.529 INFO:teuthology.orchestra.run.smithi008.stdout:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-25T02:49:44.942 INFO:teuthology.orchestra.run.smithi008.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-25T02:49:44.954 INFO:teuthology.orchestra.run.smithi008.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-25T02:49:46.986 INFO:teuthology.orchestra.run.smithi008.stdout:done 2023-01-25T02:49:47.405 DEBUG:teuthology.orchestra.run.smithi008:> dpkg -s linux-image-generic 2023-01-25T02:49:47.454 INFO:teuthology.orchestra.run.smithi008.stdout:Package: linux-image-generic 2023-01-25T02:49:47.454 INFO:teuthology.orchestra.run.smithi008.stdout:Status: install ok installed 2023-01-25T02:49:47.454 INFO:teuthology.orchestra.run.smithi008.stdout:Priority: optional 2023-01-25T02:49:47.454 INFO:teuthology.orchestra.run.smithi008.stdout:Section: kernel 2023-01-25T02:49:47.455 INFO:teuthology.orchestra.run.smithi008.stdout:Installed-Size: 19 2023-01-25T02:49:47.455 INFO:teuthology.orchestra.run.smithi008.stdout:Maintainer: Ubuntu Kernel Team 2023-01-25T02:49:47.455 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: amd64 2023-01-25T02:49:47.455 INFO:teuthology.orchestra.run.smithi008.stdout:Source: linux-meta 2023-01-25T02:49:47.455 INFO:teuthology.orchestra.run.smithi008.stdout:Version: 5.4.0.137.135 2023-01-25T02:49:47.456 INFO:teuthology.orchestra.run.smithi008.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-25T02:49:47.456 INFO:teuthology.orchestra.run.smithi008.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-25T02:49:47.456 INFO:teuthology.orchestra.run.smithi008.stdout:Recommends: thermald 2023-01-25T02:49:47.456 INFO:teuthology.orchestra.run.smithi008.stdout:Description: Generic Linux kernel image 2023-01-25T02:49:47.456 INFO:teuthology.orchestra.run.smithi008.stdout: This package will always depend on the latest generic kernel image 2023-01-25T02:49:47.457 INFO:teuthology.orchestra.run.smithi008.stdout: available. 2023-01-25T02:49:47.457 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-137-generic 2023-01-25T02:49:47.457 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T02:49:47.457 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T02:49:47.458 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-25T02:49:47.458 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T02:49:47.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get clean 2023-01-25T02:49:47.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get update 2023-01-25T02:49:47.667 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-25T02:49:47.805 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-25T02:49:47.897 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-25T02:49:47.989 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-25T02:49:49.386 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2023-01-25T02:49:49.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-25T02:49:49.480 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2023-01-25T02:49:49.670 INFO:teuthology.orchestra.run.smithi008.stdout:Building dependency tree... 2023-01-25T02:49:49.670 INFO:teuthology.orchestra.run.smithi008.stdout:Reading state information... 2023-01-25T02:49:49.889 INFO:teuthology.orchestra.run.smithi008.stdout:linux-image-generic is already the newest version (5.4.0.137.135). 2023-01-25T02:49:49.890 INFO:teuthology.orchestra.run.smithi008.stdout:0 upgraded, 0 newly installed, 0 to remove and 129 not upgraded. 2023-01-25T02:49:49.891 DEBUG:teuthology.orchestra.run.smithi008:> dpkg -s linux-image-generic 2023-01-25T02:49:49.912 INFO:teuthology.orchestra.run.smithi008.stdout:Package: linux-image-generic 2023-01-25T02:49:49.913 INFO:teuthology.orchestra.run.smithi008.stdout:Status: install ok installed 2023-01-25T02:49:49.913 INFO:teuthology.orchestra.run.smithi008.stdout:Priority: optional 2023-01-25T02:49:49.913 INFO:teuthology.orchestra.run.smithi008.stdout:Section: kernel 2023-01-25T02:49:49.913 INFO:teuthology.orchestra.run.smithi008.stdout:Installed-Size: 19 2023-01-25T02:49:49.914 INFO:teuthology.orchestra.run.smithi008.stdout:Maintainer: Ubuntu Kernel Team 2023-01-25T02:49:49.914 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: amd64 2023-01-25T02:49:49.914 INFO:teuthology.orchestra.run.smithi008.stdout:Source: linux-meta 2023-01-25T02:49:49.914 INFO:teuthology.orchestra.run.smithi008.stdout:Version: 5.4.0.137.135 2023-01-25T02:49:49.915 INFO:teuthology.orchestra.run.smithi008.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-25T02:49:49.915 INFO:teuthology.orchestra.run.smithi008.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-25T02:49:49.915 INFO:teuthology.orchestra.run.smithi008.stdout:Recommends: thermald 2023-01-25T02:49:49.915 INFO:teuthology.orchestra.run.smithi008.stdout:Description: Generic Linux kernel image 2023-01-25T02:49:49.915 INFO:teuthology.orchestra.run.smithi008.stdout: This package will always depend on the latest generic kernel image 2023-01-25T02:49:49.916 INFO:teuthology.orchestra.run.smithi008.stdout: available. 2023-01-25T02:49:49.916 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2023-01-25T02:49:49.961 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.ilBFizI2Nb 2023-01-25T02:49:49.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ilBFizI2Nb 2023-01-25T02:49:49.976 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.ilBFizI2Nb 2023-01-25T02:49:50.019 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.ilBFizI2Nb is 9KB 2023-01-25T02:49:50.034 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.ilBFizI2Nb 2023-01-25T02:49:50.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-01-25T02:49:50.099 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:49:50.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-01-25T02:49:50.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-01-25T02:49:50.146 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-137-generic 2023-01-25T02:49:50.146 DEBUG:teuthology.orchestra.run.smithi008:> sudo update-grub 2023-01-25T02:49:51.512 INFO:teuthology.orchestra.run.smithi008.stderr:Sourcing file `/etc/default/grub' 2023-01-25T02:49:51.523 INFO:teuthology.orchestra.run.smithi008.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-25T02:49:51.536 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2023-01-25T02:49:51.842 INFO:teuthology.orchestra.run.smithi008.stderr:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-25T02:49:51.853 INFO:teuthology.orchestra.run.smithi008.stderr:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-25T02:49:52.217 INFO:teuthology.orchestra.run.smithi008.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-25T02:49:52.229 INFO:teuthology.orchestra.run.smithi008.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-25T02:49:53.258 INFO:teuthology.orchestra.run.smithi008.stderr:done 2023-01-25T02:49:53.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2023-01-25T02:50:23.332 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-25T02:50:23.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2023-01-25T02:50:23.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T02:50:41.896 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-01-25T02:51:11.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2023-01-25T02:51:11.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T02:51:30.280 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-01-25T02:52:00.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2023-01-25T02:52:00.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T02:52:00.509 DEBUG:teuthology.orchestra.run.smithi008:> true 2023-01-25T02:52:01.097 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2023-01-25T02:52:01.097 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-137-generic"... 2023-01-25T02:52:01.097 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2023-01-25T02:52:01.104 INFO:teuthology.orchestra.run.smithi008.stdout:5.4.0-137-generic 2023-01-25T02:52:01.104 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-137-generic vs 5.4.0-137-generic 2023-01-25T02:52:01.104 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T02:52:01.104 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-01-25T02:52:02.105 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-25T02:52:02.106 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T02:52:02.210 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2023-01-25T02:52:02.234 DEBUG:teuthology.parallel:result is None 2023-01-25T02:52:02.235 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-25T02:52:02.273 INFO:teuthology.task.internal:Creating test directory... 2023-01-25T02:52:02.273 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T02:52:02.281 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-25T02:52:02.310 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-25T02:52:02.360 INFO:teuthology.task.internal:Creating archive directory... 2023-01-25T02:52:02.360 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T02:52:02.389 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-25T02:52:02.410 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-25T02:52:02.411 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T02:52:02.429 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T02:52:02.439 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T02:52:02.442 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-25T02:52:02.477 INFO:teuthology.task.internal:Configuring sudo... 2023-01-25T02:52:02.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T02:52:02.500 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-25T02:52:02.530 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-25T02:52:02.530 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T02:52:02.537 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T02:52:02.544 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T02:52:02.550 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:52:02.551 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T02:52:02.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2023-01-25T02:52:02.669 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-25T02:52:02.681 INFO:teuthology.task.internal:Starting timer... 2023-01-25T02:52:02.681 INFO:teuthology.run_tasks:Running task pcp... 2023-01-25T02:52:02.714 INFO:teuthology.run_tasks:Running task selinux... 2023-01-25T02:52:02.764 DEBUG:teuthology.task.selinux:Excluding smithi008: OS 'ubuntu' does not support SELinux 2023-01-25T02:52:02.765 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-25T02:52:02.765 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-01-25T02:52:02.765 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-25T02:52:02.766 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-25T02:52:02.813 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-25T02:52:02.818 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-25T02:52:02.933 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-25T02:52:02.951 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-25T02:52:02.952 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-25T02:56:39.293 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com')] 2023-01-25T02:56:39.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2023-01-25T02:56:39.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T02:56:39.344 DEBUG:teuthology.orchestra.run.smithi008:> true 2023-01-25T02:56:39.470 INFO:teuthology.run_tasks:Running task clock... 2023-01-25T02:56:39.479 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-25T02:56:39.480 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-25T02:56:39.481 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T02:56:39.516 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-01-25T02:56:39.517 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Command line: ntpd -gq 2023-01-25T02:56:39.517 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: proto: precision = 0.084 usec (-23) 2023-01-25T02:56:39.518 INFO:teuthology.orchestra.run.smithi008.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-25T02:56:39.518 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-25T02:56:39.519 INFO:teuthology.orchestra.run.smithi008.stderr:restrict ::: KOD does nothing without LIMITED. 2023-01-25T02:56:39.519 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: restrict ::: KOD does nothing without LIMITED. 2023-01-25T02:56:39.520 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listen and drop on 0 v6wildcard [::]:123 2023-01-25T02:56:39.520 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-01-25T02:56:39.520 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listen normally on 2 lo 127.0.0.1:123 2023-01-25T02:56:39.520 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listen normally on 3 ens1f0 172.21.15.8:123 2023-01-25T02:56:39.521 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listen normally on 4 lo [::1]:123 2023-01-25T02:56:39.521 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:25fe%4]:123 2023-01-25T02:56:39.521 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:39 ntpd[19309]: Listening on routing socket on fd #22 for interface updates 2023-01-25T02:56:40.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:40 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:40.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:40 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:41.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:41 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:41.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:41 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:41.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:41 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:41.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:41 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:41.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:41 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:41.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:41 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:42.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:42 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:42.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:42 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:42.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:42 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:42.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:42 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:42.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:42 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:42.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:42 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:43.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:43.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:43.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:43.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:43.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:43.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:43.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:43.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:43.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:43.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:43.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:43.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:43 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:44.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:44 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:44.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:44 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:44.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:44 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:44.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:44 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:44.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:44 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:44.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:44 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:45.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:45.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:45.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:45.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:45.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:45.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:45.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:45.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:45.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:45.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:45.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:45.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:45 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:46.517 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:46 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:46.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:46 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:46.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:46 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:46.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:46 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:46.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:46 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:46.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:46 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:47.518 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T02:56:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T02:56:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T02:56:47.520 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T02:56:47.521 INFO:teuthology.orchestra.run.smithi008.stdout:25 Jan 02:56:47 ntpd[19309]: ntpd: time slew +0.000345 s 2023-01-25T02:56:47.521 INFO:teuthology.orchestra.run.smithi008.stdout:ntpd: time slew +0.000345s 2023-01-25T02:56:47.522 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-01-25T02:56:47.522 INFO:teuthology.orchestra.run.smithi008.stderr:25 Jan 02:56:47 ntpd[19309]: can't open /var/log/ntpstats/loopstats.20230125: Permission denied 2023-01-25T02:56:47.577 INFO:teuthology.orchestra.run.smithi008.stdout: remote refid st t when poll reach delay offset jitter 2023-01-25T02:56:47.577 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================== 2023-01-25T02:56:47.577 INFO:teuthology.orchestra.run.smithi008.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T02:56:47.577 INFO:teuthology.orchestra.run.smithi008.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T02:56:47.578 INFO:teuthology.orchestra.run.smithi008.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T02:56:47.578 INFO:teuthology.orchestra.run.smithi008.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T02:56:47.579 INFO:teuthology.run_tasks:Running task install... 2023-01-25T02:56:47.587 DEBUG:teuthology.task.install:project ceph 2023-01-25T02:56:47.588 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'}} 2023-01-25T02:56:47.588 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'} 2023-01-25T02:56:47.588 INFO:teuthology.task.install:Using flavor: default 2023-01-25T02:56:47.595 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-25T02:56:47.596 INFO:teuthology.task.install:extra packages: [] 2023-01-25T02:56:47.596 DEBUG:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2023-01-25T02:56:47.668 INFO:teuthology.orchestra.run.smithi008.stdout:Ubuntu 2023-01-25T02:56:47.674 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-25T02:56:47.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-key list | grep Ceph 2023-01-25T02:56:47.780 INFO:teuthology.orchestra.run.smithi008.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-01-25T02:56:47.842 INFO:teuthology.orchestra.run.smithi008.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-01-25T02:56:47.842 INFO:teuthology.orchestra.run.smithi008.stdout:uid [ unknown] Ceph.com (release key) 2023-01-25T02:56:47.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2023-01-25T02:56:47.893 INFO:teuthology.orchestra.run.smithi008.stdout:Ubuntu 2023-01-25T02:56:47.898 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-25T02:56:47.899 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-01-25T02:56:47.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T02:56:48.056 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default/ 2023-01-25T02:56:48.057 INFO:teuthology.task.install.deb:Package version is 18.0.0-1963-g61996d73-1focal 2023-01-25T02:56:48.172 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:56:48.172 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-01-25T02:56:48.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo apt-get update 2023-01-25T02:56:48.333 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-25T02:56:48.405 INFO:teuthology.orchestra.run.smithi008.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal InRelease 2023-01-25T02:56:48.415 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-25T02:56:48.431 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-01-25T02:56:48.432 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-25T02:56:48.450 INFO:teuthology.orchestra.run.smithi008.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-25T02:56:48.456 INFO:teuthology.orchestra.run.smithi008.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal Release.gpg 2023-01-25T02:56:48.495 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main i386 Packages [3,428 B] 2023-01-25T02:56:48.555 INFO:teuthology.orchestra.run.smithi008.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-01-25T02:56:49.008 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 28.4 kB in 1s (43.9 kB/s) 2023-01-25T02:56:49.980 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2023-01-25T02:56:50.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-1963-g61996d73-1focal cephadm=18.0.0-1963-g61996d73-1focal ceph-mds=18.0.0-1963-g61996d73-1focal ceph-mgr=18.0.0-1963-g61996d73-1focal ceph-common=18.0.0-1963-g61996d73-1focal ceph-fuse=18.0.0-1963-g61996d73-1focal ceph-test=18.0.0-1963-g61996d73-1focal ceph-volume=18.0.0-1963-g61996d73-1focal radosgw=18.0.0-1963-g61996d73-1focal python3-rados=18.0.0-1963-g61996d73-1focal python3-rgw=18.0.0-1963-g61996d73-1focal python3-cephfs=18.0.0-1963-g61996d73-1focal python3-rbd=18.0.0-1963-g61996d73-1focal libcephfs2=18.0.0-1963-g61996d73-1focal libcephfs-dev=18.0.0-1963-g61996d73-1focal librados2=18.0.0-1963-g61996d73-1focal librbd1=18.0.0-1963-g61996d73-1focal rbd-fuse=18.0.0-1963-g61996d73-1focal 2023-01-25T02:56:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:Reading package lists... 2023-01-25T02:56:50.263 INFO:teuthology.orchestra.run.smithi008.stdout:Building dependency tree... 2023-01-25T02:56:50.264 INFO:teuthology.orchestra.run.smithi008.stdout:Reading state information... 2023-01-25T02:56:50.439 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages were automatically installed and are no longer required: 2023-01-25T02:56:50.440 INFO:teuthology.orchestra.run.smithi008.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-25T02:56:50.440 INFO:teuthology.orchestra.run.smithi008.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-25T02:56:50.440 INFO:teuthology.orchestra.run.smithi008.stdout:The following additional packages will be installed: 2023-01-25T02:56:50.441 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-01-25T02:56:50.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-01-25T02:56:50.442 INFO:teuthology.orchestra.run.smithi008.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-01-25T02:56:50.442 INFO:teuthology.orchestra.run.smithi008.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-01-25T02:56:50.442 INFO:teuthology.orchestra.run.smithi008.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-01-25T02:56:50.443 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-01-25T02:56:50.443 INFO:teuthology.orchestra.run.smithi008.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-01-25T02:56:50.443 INFO:teuthology.orchestra.run.smithi008.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-01-25T02:56:50.443 INFO:teuthology.orchestra.run.smithi008.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-01-25T02:56:50.443 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-01-25T02:56:50.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-01-25T02:56:50.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-25T02:56:50.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-25T02:56:50.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-25T02:56:50.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-25T02:56:50.445 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-25T02:56:50.445 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-01-25T02:56:50.445 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-01-25T02:56:50.445 INFO:teuthology.orchestra.run.smithi008.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-01-25T02:56:50.445 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-01-25T02:56:50.446 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-01-25T02:56:50.446 INFO:teuthology.orchestra.run.smithi008.stdout: qttranslations5-l10n socat xmlstarlet 2023-01-25T02:56:50.446 INFO:teuthology.orchestra.run.smithi008.stdout:Suggested packages: 2023-01-25T02:56:50.446 INFO:teuthology.orchestra.run.smithi008.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-01-25T02:56:50.447 INFO:teuthology.orchestra.run.smithi008.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-01-25T02:56:50.447 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-01-25T02:56:50.447 INFO:teuthology.orchestra.run.smithi008.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-01-25T02:56:50.447 INFO:teuthology.orchestra.run.smithi008.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-01-25T02:56:50.447 INFO:teuthology.orchestra.run.smithi008.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-01-25T02:56:50.448 INFO:teuthology.orchestra.run.smithi008.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-01-25T02:56:50.448 INFO:teuthology.orchestra.run.smithi008.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-01-25T02:56:50.448 INFO:teuthology.orchestra.run.smithi008.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-01-25T02:56:50.448 INFO:teuthology.orchestra.run.smithi008.stdout:Recommended packages: 2023-01-25T02:56:50.448 INFO:teuthology.orchestra.run.smithi008.stdout: btrfs-tools 2023-01-25T02:56:50.558 INFO:teuthology.orchestra.run.smithi008.stdout:The following NEW packages will be installed: 2023-01-25T02:56:50.558 INFO:teuthology.orchestra.run.smithi008.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-01-25T02:56:50.558 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-01-25T02:56:50.559 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-01-25T02:56:50.559 INFO:teuthology.orchestra.run.smithi008.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-01-25T02:56:50.559 INFO:teuthology.orchestra.run.smithi008.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-01-25T02:56:50.559 INFO:teuthology.orchestra.run.smithi008.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-01-25T02:56:50.559 INFO:teuthology.orchestra.run.smithi008.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-01-25T02:56:50.561 INFO:teuthology.orchestra.run.smithi008.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-01-25T02:56:50.561 INFO:teuthology.orchestra.run.smithi008.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-01-25T02:56:50.561 INFO:teuthology.orchestra.run.smithi008.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-01-25T02:56:50.562 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-01-25T02:56:50.562 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-01-25T02:56:50.562 INFO:teuthology.orchestra.run.smithi008.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-01-25T02:56:50.562 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-25T02:56:50.563 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-25T02:56:50.563 INFO:teuthology.orchestra.run.smithi008.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-25T02:56:50.563 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-25T02:56:50.563 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-25T02:56:50.563 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-01-25T02:56:50.564 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-01-25T02:56:50.564 INFO:teuthology.orchestra.run.smithi008.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-01-25T02:56:50.564 INFO:teuthology.orchestra.run.smithi008.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-01-25T02:56:50.564 INFO:teuthology.orchestra.run.smithi008.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-01-25T02:56:50.564 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-01-25T02:56:50.565 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet 2023-01-25T02:56:50.565 INFO:teuthology.orchestra.run.smithi008.stdout:The following packages will be upgraded: 2023-01-25T02:56:50.565 INFO:teuthology.orchestra.run.smithi008.stdout: librados2 librbd1 2023-01-25T02:56:50.734 INFO:teuthology.orchestra.run.smithi008.stdout:2 upgraded, 110 newly installed, 0 to remove and 124 not upgraded. 2023-01-25T02:56:50.734 INFO:teuthology.orchestra.run.smithi008.stdout:Need to get 210 MB of archives. 2023-01-25T02:56:50.734 INFO:teuthology.orchestra.run.smithi008.stdout:After this operation, 1,029 MB of additional disk space will be used. 2023-01-25T02:56:50.734 INFO:teuthology.orchestra.run.smithi008.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.0.0-1963-g61996d73-1focal [3,428 kB] 2023-01-25T02:56:50.770 INFO:teuthology.orchestra.run.smithi008.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-01-25T02:56:50.982 INFO:teuthology.orchestra.run.smithi008.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.0.0-1963-g61996d73-1focal [3,963 kB] 2023-01-25T02:56:51.116 INFO:teuthology.orchestra.run.smithi008.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.0.0-1963-g61996d73-1focal [719 kB] 2023-01-25T02:56:51.167 INFO:teuthology.orchestra.run.smithi008.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.0.0-1963-g61996d73-1focal [376 kB] 2023-01-25T02:56:51.206 INFO:teuthology.orchestra.run.smithi008.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.0.0-1963-g61996d73-1focal [22.4 kB] 2023-01-25T02:56:51.212 INFO:teuthology.orchestra.run.smithi008.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-01-25T02:56:51.228 INFO:teuthology.orchestra.run.smithi008.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-01-25T02:56:51.241 INFO:teuthology.orchestra.run.smithi008.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.0.0-1963-g61996d73-1focal [184 kB] 2023-01-25T02:56:51.245 INFO:teuthology.orchestra.run.smithi008.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.0.0-1963-g61996d73-1focal [46.5 kB] 2023-01-25T02:56:51.247 INFO:teuthology.orchestra.run.smithi008.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.0.0-1963-g61996d73-1focal [354 kB] 2023-01-25T02:56:51.271 INFO:teuthology.orchestra.run.smithi008.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.0.0-1963-g61996d73-1focal [6,516 kB] 2023-01-25T02:56:51.321 INFO:teuthology.orchestra.run.smithi008.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-01-25T02:56:51.508 INFO:teuthology.orchestra.run.smithi008.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.0.0-1963-g61996d73-1focal [104 kB] 2023-01-25T02:56:51.511 INFO:teuthology.orchestra.run.smithi008.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.0.0-1963-g61996d73-1focal [469 kB] 2023-01-25T02:56:51.526 INFO:teuthology.orchestra.run.smithi008.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.0.0-1963-g61996d73-1focal [29.6 MB] 2023-01-25T02:56:51.592 INFO:teuthology.orchestra.run.smithi008.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-01-25T02:56:51.603 INFO:teuthology.orchestra.run.smithi008.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-01-25T02:56:51.635 INFO:teuthology.orchestra.run.smithi008.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-01-25T02:56:51.645 INFO:teuthology.orchestra.run.smithi008.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-01-25T02:56:51.646 INFO:teuthology.orchestra.run.smithi008.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-01-25T02:56:51.650 INFO:teuthology.orchestra.run.smithi008.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-01-25T02:56:51.651 INFO:teuthology.orchestra.run.smithi008.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-01-25T02:56:51.665 INFO:teuthology.orchestra.run.smithi008.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-01-25T02:56:51.666 INFO:teuthology.orchestra.run.smithi008.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-01-25T02:56:51.683 INFO:teuthology.orchestra.run.smithi008.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-01-25T02:56:51.762 INFO:teuthology.orchestra.run.smithi008.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-01-25T02:56:51.762 INFO:teuthology.orchestra.run.smithi008.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-01-25T02:56:51.763 INFO:teuthology.orchestra.run.smithi008.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-01-25T02:56:51.765 INFO:teuthology.orchestra.run.smithi008.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-01-25T02:56:51.766 INFO:teuthology.orchestra.run.smithi008.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-01-25T02:56:51.766 INFO:teuthology.orchestra.run.smithi008.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-01-25T02:56:51.768 INFO:teuthology.orchestra.run.smithi008.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-01-25T02:56:51.769 INFO:teuthology.orchestra.run.smithi008.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-01-25T02:56:51.774 INFO:teuthology.orchestra.run.smithi008.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-01-25T02:56:51.863 INFO:teuthology.orchestra.run.smithi008.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-01-25T02:56:51.863 INFO:teuthology.orchestra.run.smithi008.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-01-25T02:56:51.872 INFO:teuthology.orchestra.run.smithi008.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-01-25T02:56:51.873 INFO:teuthology.orchestra.run.smithi008.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-01-25T02:56:51.874 INFO:teuthology.orchestra.run.smithi008.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-01-25T02:56:51.875 INFO:teuthology.orchestra.run.smithi008.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-01-25T02:56:51.876 INFO:teuthology.orchestra.run.smithi008.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2023-01-25T02:56:51.878 INFO:teuthology.orchestra.run.smithi008.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-01-25T02:56:51.880 INFO:teuthology.orchestra.run.smithi008.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-01-25T02:56:51.957 INFO:teuthology.orchestra.run.smithi008.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-01-25T02:56:51.963 INFO:teuthology.orchestra.run.smithi008.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-01-25T02:56:52.052 INFO:teuthology.orchestra.run.smithi008.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-01-25T02:56:52.053 INFO:teuthology.orchestra.run.smithi008.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-01-25T02:56:52.054 INFO:teuthology.orchestra.run.smithi008.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-01-25T02:56:52.055 INFO:teuthology.orchestra.run.smithi008.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-01-25T02:56:52.056 INFO:teuthology.orchestra.run.smithi008.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-01-25T02:56:52.058 INFO:teuthology.orchestra.run.smithi008.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-01-25T02:56:52.059 INFO:teuthology.orchestra.run.smithi008.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-01-25T02:56:52.060 INFO:teuthology.orchestra.run.smithi008.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-01-25T02:56:52.143 INFO:teuthology.orchestra.run.smithi008.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-01-25T02:56:52.146 INFO:teuthology.orchestra.run.smithi008.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-01-25T02:56:52.233 INFO:teuthology.orchestra.run.smithi008.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-01-25T02:56:52.236 INFO:teuthology.orchestra.run.smithi008.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-01-25T02:56:52.238 INFO:teuthology.orchestra.run.smithi008.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-01-25T02:56:52.240 INFO:teuthology.orchestra.run.smithi008.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-01-25T02:56:52.241 INFO:teuthology.orchestra.run.smithi008.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-01-25T02:56:52.242 INFO:teuthology.orchestra.run.smithi008.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-01-25T02:56:52.779 INFO:teuthology.orchestra.run.smithi008.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-01-25T02:56:52.779 INFO:teuthology.orchestra.run.smithi008.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.0.0-1963-g61996d73-1focal [10.7 MB] 2023-01-25T02:56:52.805 INFO:teuthology.orchestra.run.smithi008.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-01-25T02:56:52.806 INFO:teuthology.orchestra.run.smithi008.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-01-25T02:56:52.881 INFO:teuthology.orchestra.run.smithi008.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-01-25T02:56:52.881 INFO:teuthology.orchestra.run.smithi008.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-01-25T02:56:52.882 INFO:teuthology.orchestra.run.smithi008.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-01-25T02:56:52.883 INFO:teuthology.orchestra.run.smithi008.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-01-25T02:56:52.884 INFO:teuthology.orchestra.run.smithi008.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-01-25T02:56:52.885 INFO:teuthology.orchestra.run.smithi008.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-01-25T02:56:52.885 INFO:teuthology.orchestra.run.smithi008.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-01-25T02:56:52.961 INFO:teuthology.orchestra.run.smithi008.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-01-25T02:56:52.962 INFO:teuthology.orchestra.run.smithi008.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-01-25T02:56:53.011 INFO:teuthology.orchestra.run.smithi008.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.0.0-1963-g61996d73-1focal [221 kB] 2023-01-25T02:56:53.017 INFO:teuthology.orchestra.run.smithi008.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.0.0-1963-g61996d73-1focal [133 kB] 2023-01-25T02:56:53.021 INFO:teuthology.orchestra.run.smithi008.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.0.0-1963-g61996d73-1focal [1,533 kB] 2023-01-25T02:56:53.052 INFO:teuthology.orchestra.run.smithi008.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-01-25T02:56:53.057 INFO:teuthology.orchestra.run.smithi008.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-01-25T02:56:53.060 INFO:teuthology.orchestra.run.smithi008.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-01-25T02:56:53.063 INFO:teuthology.orchestra.run.smithi008.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-01-25T02:56:53.064 INFO:teuthology.orchestra.run.smithi008.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-01-25T02:56:53.067 INFO:teuthology.orchestra.run.smithi008.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-01-25T02:56:53.069 INFO:teuthology.orchestra.run.smithi008.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-01-25T02:56:53.099 INFO:teuthology.orchestra.run.smithi008.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.0.0-1963-g61996d73-1focal [7,857 kB] 2023-01-25T02:56:53.148 INFO:teuthology.orchestra.run.smithi008.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-01-25T02:56:53.152 INFO:teuthology.orchestra.run.smithi008.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-01-25T02:56:53.156 INFO:teuthology.orchestra.run.smithi008.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-01-25T02:56:53.329 INFO:teuthology.orchestra.run.smithi008.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-01-25T02:56:53.330 INFO:teuthology.orchestra.run.smithi008.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-01-25T02:56:53.331 INFO:teuthology.orchestra.run.smithi008.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-01-25T02:56:53.333 INFO:teuthology.orchestra.run.smithi008.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-01-25T02:56:53.350 INFO:teuthology.orchestra.run.smithi008.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.0.0-1963-g61996d73-1focal [28.9 MB] 2023-01-25T02:56:53.381 INFO:teuthology.orchestra.run.smithi008.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-01-25T02:56:53.382 INFO:teuthology.orchestra.run.smithi008.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-01-25T02:56:53.385 INFO:teuthology.orchestra.run.smithi008.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-01-25T02:56:53.386 INFO:teuthology.orchestra.run.smithi008.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-01-25T02:56:53.386 INFO:teuthology.orchestra.run.smithi008.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-01-25T02:56:54.405 INFO:teuthology.orchestra.run.smithi008.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.0.0-1963-g61996d73-1focal [3,808 B] 2023-01-25T02:56:54.476 INFO:teuthology.orchestra.run.smithi008.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.0.0-1963-g61996d73-1focal [829 kB] 2023-01-25T02:56:54.490 INFO:teuthology.orchestra.run.smithi008.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.0.0-1963-g61996d73-1focal [2,279 kB] 2023-01-25T02:56:54.506 INFO:teuthology.orchestra.run.smithi008.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.0.0-1963-g61996d73-1focal [96.2 kB] 2023-01-25T02:56:54.507 INFO:teuthology.orchestra.run.smithi008.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.0.0-1963-g61996d73-1focal [115 kB] 2023-01-25T02:56:54.508 INFO:teuthology.orchestra.run.smithi008.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.0.0-1963-g61996d73-1focal [1,787 kB] 2023-01-25T02:56:54.525 INFO:teuthology.orchestra.run.smithi008.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.0.0-1963-g61996d73-1focal [7,891 kB] 2023-01-25T02:56:54.787 INFO:teuthology.orchestra.run.smithi008.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.0.0-1963-g61996d73-1focal [3,928 B] 2023-01-25T02:56:54.788 INFO:teuthology.orchestra.run.smithi008.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.0.0-1963-g61996d73-1focal [53.4 MB] 2023-01-25T02:56:57.578 INFO:teuthology.orchestra.run.smithi008.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.0.0-1963-g61996d73-1focal [225 kB] 2023-01-25T02:56:57.582 INFO:teuthology.orchestra.run.smithi008.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.0.0-1963-g61996d73-1focal [27.4 kB] 2023-01-25T02:56:57.808 INFO:teuthology.orchestra.run.smithi008.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.0.0-1963-g61996d73-1focal [13.3 MB] 2023-01-25T02:56:58.512 INFO:teuthology.orchestra.run.smithi008.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.0.0-1963-g61996d73-1focal [97.2 kB] 2023-01-25T02:56:59.278 INFO:teuthology.orchestra.run.smithi008.stdout:Fetched 210 MB in 8s (26.0 MB/s) 2023-01-25T02:56:59.577 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-01-25T02:56:59.635 INFO:teuthology.orchestra.run.smithi008.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141754 files and directories currently installed.) 2023-01-25T02:56:59.639 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-01-25T02:56:59.726 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-25T02:57:00.080 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-01-25T02:57:00.096 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-01-25T02:57:00.129 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-25T02:57:00.416 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-01-25T02:57:00.434 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-01-25T02:57:00.474 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-25T02:57:00.953 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-01-25T02:57:00.973 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-25T02:57:01.021 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T02:57:01.531 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-01-25T02:57:01.550 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-25T02:57:01.581 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T02:57:01.943 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-01-25T02:57:01.959 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-25T02:57:02.001 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T02:57:02.554 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-01-25T02:57:02.571 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-01-25T02:57:02.612 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-25T02:57:02.943 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../007-librbd1_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:03.041 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librbd1 (18.0.0-1963-g61996d73-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-25T02:57:03.745 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../008-librados2_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:03.870 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librados2 (18.0.0-1963-g61996d73-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-25T02:57:04.616 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libcephfs2. 2023-01-25T02:57:04.632 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../009-libcephfs2_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:04.675 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libcephfs2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:05.127 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rados. 2023-01-25T02:57:05.143 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../010-python3-rados_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:05.177 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rados (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:05.497 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-01-25T02:57:05.513 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:05.563 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-ceph-argparse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:05.808 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cephfs. 2023-01-25T02:57:05.824 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../012-python3-cephfs_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:05.866 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cephfs (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:06.219 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-ceph-common. 2023-01-25T02:57:06.235 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../013-python3-ceph-common_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:06.277 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:06.564 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-prettytable. 2023-01-25T02:57:06.582 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-01-25T02:57:06.622 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-01-25T02:57:06.858 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rbd. 2023-01-25T02:57:06.875 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../015-python3-rbd_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:06.916 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rbd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:07.297 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-01-25T02:57:07.315 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-01-25T02:57:07.381 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-25T02:57:07.728 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-01-25T02:57:07.744 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-01-25T02:57:07.787 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-01-25T02:57:08.323 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-01-25T02:57:08.342 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-01-25T02:57:08.374 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-25T02:57:08.710 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package librgw2. 2023-01-25T02:57:08.730 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../019-librgw2_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:08.768 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking librgw2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:09.620 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rgw. 2023-01-25T02:57:09.638 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:09.679 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:10.057 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liboath0. 2023-01-25T02:57:10.077 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-01-25T02:57:10.118 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-01-25T02:57:10.454 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libradosstriper1. 2023-01-25T02:57:10.471 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:10.512 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libradosstriper1 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:10.966 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-common. 2023-01-25T02:57:10.983 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../023-ceph-common_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:11.040 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:14.054 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-base. 2023-01-25T02:57:14.071 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../024-ceph-base_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:14.123 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-base (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:15.517 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cherrypy3. 2023-01-25T02:57:15.533 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-01-25T02:57:15.567 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-01-25T02:57:15.855 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-natsort. 2023-01-25T02:57:15.873 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-01-25T02:57:15.904 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-01-25T02:57:16.224 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-logutils. 2023-01-25T02:57:16.242 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-01-25T02:57:16.274 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-01-25T02:57:16.618 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-markupsafe. 2023-01-25T02:57:16.638 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-01-25T02:57:16.676 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-01-25T02:57:16.979 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-mako. 2023-01-25T02:57:16.996 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-01-25T02:57:17.037 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-25T02:57:17.391 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-simplegeneric. 2023-01-25T02:57:17.410 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-01-25T02:57:17.449 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-01-25T02:57:17.744 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-singledispatch. 2023-01-25T02:57:17.762 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-01-25T02:57:17.810 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-01-25T02:57:18.113 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-webob. 2023-01-25T02:57:18.129 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-01-25T02:57:18.171 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-01-25T02:57:18.541 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-soupsieve. 2023-01-25T02:57:18.557 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-01-25T02:57:18.599 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-25T02:57:18.894 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-bs4. 2023-01-25T02:57:18.913 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-01-25T02:57:18.952 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-01-25T02:57:19.367 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-waitress. 2023-01-25T02:57:19.383 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-01-25T02:57:19.428 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-25T02:57:19.778 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-tempita. 2023-01-25T02:57:19.798 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-01-25T02:57:19.836 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-01-25T02:57:20.139 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-paste. 2023-01-25T02:57:20.156 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-01-25T02:57:20.198 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-01-25T02:57:20.517 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-01-25T02:57:20.535 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-01-25T02:57:20.576 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-01-25T02:57:20.904 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pastedeploy. 2023-01-25T02:57:20.922 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-01-25T02:57:20.962 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-01-25T02:57:21.274 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-webtest. 2023-01-25T02:57:21.291 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-01-25T02:57:21.332 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-01-25T02:57:21.668 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pecan. 2023-01-25T02:57:21.684 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-01-25T02:57:21.735 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-01-25T02:57:22.048 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-werkzeug. 2023-01-25T02:57:22.064 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2023-01-25T02:57:22.123 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-25T02:57:22.410 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-01-25T02:57:22.427 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:22.468 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-modules-core (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:22.873 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-01-25T02:57:22.893 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:22.931 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:23.251 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cryptography. 2023-01-25T02:57:23.267 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-01-25T02:57:23.334 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-25T02:57:24.380 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-openssl. 2023-01-25T02:57:24.396 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-01-25T02:57:24.438 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-01-25T02:57:24.808 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr. 2023-01-25T02:57:24.824 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../047-ceph-mgr_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:24.866 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:25.378 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mon. 2023-01-25T02:57:25.395 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../048-ceph-mon_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:25.436 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mon (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:26.698 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-osd. 2023-01-25T02:57:26.718 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../049-ceph-osd_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:26.756 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-osd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:29.818 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph. 2023-01-25T02:57:29.835 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../050-ceph_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:29.876 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:30.229 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-fuse. 2023-01-25T02:57:30.245 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../051-ceph-fuse_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:30.287 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:30.690 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mds. 2023-01-25T02:57:30.707 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../052-ceph-mds_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:30.740 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mds (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:31.235 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package cephadm. 2023-01-25T02:57:31.254 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../053-cephadm_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:31.310 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:31.646 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-asyncssh. 2023-01-25T02:57:31.663 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-01-25T02:57:31.713 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-01-25T02:57:32.157 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jinja2. 2023-01-25T02:57:32.177 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-01-25T02:57:32.215 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-01-25T02:57:32.552 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-01-25T02:57:32.570 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:32.618 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:32.946 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-bcrypt. 2023-01-25T02:57:32.966 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-01-25T02:57:33.004 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-25T02:57:33.316 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-jwt. 2023-01-25T02:57:33.332 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-01-25T02:57:33.374 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-25T02:57:33.719 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-attr. 2023-01-25T02:57:33.735 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-01-25T02:57:33.777 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-01-25T02:57:34.105 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pluggy. 2023-01-25T02:57:34.121 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-01-25T02:57:34.163 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-01-25T02:57:34.474 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-py. 2023-01-25T02:57:34.494 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-01-25T02:57:34.532 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-01-25T02:57:34.844 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-atomicwrites. 2023-01-25T02:57:34.862 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-01-25T02:57:34.927 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-01-25T02:57:35.205 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-packaging. 2023-01-25T02:57:35.221 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-01-25T02:57:35.255 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-packaging (20.3-1) ... 2023-01-25T02:57:35.541 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-wcwidth. 2023-01-25T02:57:35.557 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-01-25T02:57:35.591 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-25T02:57:35.877 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pytest. 2023-01-25T02:57:35.893 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-01-25T02:57:35.935 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-01-25T02:57:36.247 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-repoze.lru. 2023-01-25T02:57:36.266 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-01-25T02:57:36.297 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-01-25T02:57:36.550 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libjs-underscore. 2023-01-25T02:57:36.567 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-01-25T02:57:36.608 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-25T02:57:36.911 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-01-25T02:57:36.928 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-01-25T02:57:36.969 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-25T02:57:37.297 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-routes. 2023-01-25T02:57:37.315 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-01-25T02:57:37.371 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-01-25T02:57:37.666 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-01-25T02:57:37.686 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:37.724 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-dashboard (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:38.253 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-decorator. 2023-01-25T02:57:38.273 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-01-25T02:57:38.311 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-01-25T02:57:38.614 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-01-25T02:57:38.634 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-01-25T02:57:38.672 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-25T02:57:39.023 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-scipy. 2023-01-25T02:57:39.039 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-01-25T02:57:39.080 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-01-25T02:57:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-01-25T02:57:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-01-25T02:57:40.681 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-25T02:57:41.262 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-joblib. 2023-01-25T02:57:41.279 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-01-25T02:57:41.320 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-01-25T02:57:41.725 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-sklearn. 2023-01-25T02:57:41.743 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-01-25T02:57:41.783 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-25T02:57:42.426 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-01-25T02:57:42.444 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:42.476 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:43.696 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-cachetools. 2023-01-25T02:57:43.715 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-01-25T02:57:43.754 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-01-25T02:57:44.082 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pyasn1. 2023-01-25T02:57:44.098 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-01-25T02:57:44.140 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-01-25T02:57:44.468 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-01-25T02:57:44.486 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-01-25T02:57:44.526 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-25T02:57:44.932 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-rsa. 2023-01-25T02:57:44.948 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-01-25T02:57:44.990 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-rsa (4.0-3) ... 2023-01-25T02:57:45.327 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-google-auth. 2023-01-25T02:57:45.344 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-01-25T02:57:45.385 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-01-25T02:57:45.706 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-blinker. 2023-01-25T02:57:45.726 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-01-25T02:57:45.764 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-25T02:57:46.092 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-oauthlib. 2023-01-25T02:57:46.110 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-01-25T02:57:46.150 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-25T02:57:46.504 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-01-25T02:57:46.521 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-01-25T02:57:46.554 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-25T02:57:46.849 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-websocket. 2023-01-25T02:57:46.867 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-01-25T02:57:46.907 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-01-25T02:57:47.262 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-kubernetes. 2023-01-25T02:57:47.280 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-01-25T02:57:47.320 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-01-25T02:57:47.876 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-01-25T02:57:47.893 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:47.934 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:48.295 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libonig5:amd64. 2023-01-25T02:57:48.317 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-01-25T02:57:48.352 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-01-25T02:57:48.697 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libjq1:amd64. 2023-01-25T02:57:48.714 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-25T02:57:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-25T02:57:49.059 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package jq. 2023-01-25T02:57:49.076 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-25T02:57:49.125 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-01-25T02:57:49.386 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package socat. 2023-01-25T02:57:49.404 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-01-25T02:57:49.444 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking socat (1.7.3.3-2) ... 2023-01-25T02:57:49.757 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package xmlstarlet. 2023-01-25T02:57:49.775 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-01-25T02:57:49.806 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-01-25T02:57:50.194 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-test. 2023-01-25T02:57:50.210 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../094-ceph-test_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:50.252 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-test (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:55.140 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package ceph-volume. 2023-01-25T02:57:55.156 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../095-ceph-volume_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T02:57:55.198 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking ceph-volume (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:55.502 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package fonts-mathjax. 2023-01-25T02:57:55.520 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-25T02:57:55.561 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-25T02:57:56.106 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libcephfs-dev. 2023-01-25T02:57:56.124 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../097-libcephfs-dev_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:57:56.172 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libcephfs-dev (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:57:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package nvme-cli. 2023-01-25T02:57:56.617 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-01-25T02:57:56.676 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-01-25T02:57:57.048 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-01-25T02:57:57.065 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-01-25T02:57:57.106 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-01-25T02:57:57.509 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package libjs-mathjax. 2023-01-25T02:57:57.525 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-25T02:57:57.567 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-25T02:57:59.295 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python-natsort-doc. 2023-01-25T02:57:59.317 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-01-25T02:57:59.353 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-01-25T02:57:59.664 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-webencodings. 2023-01-25T02:57:59.684 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-01-25T02:57:59.714 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-25T02:58:00.001 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-html5lib. 2023-01-25T02:58:00.018 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-01-25T02:58:00.059 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-01-25T02:58:00.404 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-01-25T02:58:00.421 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-01-25T02:58:00.462 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-25T02:58:00.882 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pastescript. 2023-01-25T02:58:00.900 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-01-25T02:58:00.940 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-01-25T02:58:01.302 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-psutil. 2023-01-25T02:58:01.319 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-01-25T02:58:01.360 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-01-25T02:58:01.688 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-pyinotify. 2023-01-25T02:58:01.706 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-01-25T02:58:01.746 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-25T02:58:02.041 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package python3-simplejson. 2023-01-25T02:58:02.058 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-01-25T02:58:02.099 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-25T02:58:02.393 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-01-25T02:58:02.411 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-01-25T02:58:02.451 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-25T02:58:03.155 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package radosgw. 2023-01-25T02:58:03.173 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../110-radosgw_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:58:03.213 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking radosgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:04.592 INFO:teuthology.orchestra.run.smithi008.stdout:Selecting previously unselected package rbd-fuse. 2023-01-25T02:58:04.611 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing to unpack .../111-rbd-fuse_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T02:58:04.650 INFO:teuthology.orchestra.run.smithi008.stdout:Unpacking rbd-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:04.988 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-attr (19.3.0-2) ... 2023-01-25T02:58:05.338 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-01-25T02:58:05.464 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-25T02:58:05.565 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-01-25T02:58:05.733 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:06.369 INFO:teuthology.orchestra.run.smithi008.stdout:Adding system user cephadm....done 2023-01-25T02:58:06.522 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-25T02:58:06.657 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-25T02:58:06.992 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-01-25T02:58:07.285 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-25T02:58:07.410 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-01-25T02:58:07.536 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-01-25T02:58:07.871 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-01-25T02:58:08.290 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-25T02:58:08.567 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-01-25T02:58:08.851 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-25T02:58:09.127 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-25T02:58:09.420 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-ceph-argparse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:09.721 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-25T02:58:09.847 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-01-25T02:58:10.123 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-25T02:58:10.483 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-25T02:58:11.101 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-01-25T02:58:11.477 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-01-25T02:58:11.795 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-01-25T02:58:12.079 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-25T02:58:12.439 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-01-25T02:58:12.748 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-01-25T02:58:13.074 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-01-25T02:58:13.376 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-packaging (20.3-1) ... 2023-01-25T02:58:13.677 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-01-25T02:58:13.803 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-25T02:58:13.954 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-25T02:58:14.238 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-01-25T02:58:14.640 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-25T02:58:14.983 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-25T02:58:15.343 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up socat (1.7.3.3-2) ... 2023-01-25T02:58:15.469 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:15.804 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T02:58:15.938 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-01-25T02:58:16.290 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-25T02:58:16.416 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-01-25T02:58:16.525 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-01-25T02:58:16.810 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-25T02:58:17.128 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T02:58:17.237 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-25T02:58:17.363 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-01-25T02:58:17.689 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-01-25T02:58:17.982 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-01-25T02:58:18.284 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-25T02:58:18.410 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-01-25T02:58:18.694 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-01-25T02:58:18.978 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-25T02:58:19.288 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-01-25T02:58:19.647 INFO:teuthology.orchestra.run.smithi008.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-01-25T02:58:19.924 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-01-25T02:58:20.067 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-25T02:58:20.368 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-01-25T02:58:20.694 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-25T02:58:20.820 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-25T02:58:20.946 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-01-25T02:58:21.281 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-25T02:58:21.590 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-01-25T02:58:23.121 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-25T02:58:23.247 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-01-25T02:58:23.615 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-01-25T02:58:23.942 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-01-25T02:58:24.193 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T02:58:24.319 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-01-25T02:58:24.653 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-25T02:58:24.980 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-01-25T02:58:25.357 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-25T02:58:25.691 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-25T02:58:26.788 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-25T02:58:26.914 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-01-25T02:58:27.040 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-01-25T02:58:27.166 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rsa (4.0-3) ... 2023-01-25T02:58:27.459 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-25T02:58:27.560 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-01-25T02:58:27.861 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-25T02:58:28.179 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-01-25T02:58:28.489 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-01-25T02:58:28.874 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librados2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:29.000 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-01-25T02:58:29.101 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librgw2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:29.227 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libsqlite3-mod-ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:29.448 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libcephfs2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:29.574 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-routes (2.4.1-2) ... 2023-01-25T02:58:29.875 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libradosstriper1 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:29.994 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-01-25T02:58:30.298 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up librbd1 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:30.449 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-modules-core (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:30.575 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:30.805 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-25T02:58:30.805 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-25T02:58:31.268 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up libcephfs-dev (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:31.403 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rados (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:31.544 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-01-25T02:58:32.515 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rbd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:32.641 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up rbd-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:32.790 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-rgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:32.926 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up python3-cephfs (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:33.069 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:33.656 INFO:teuthology.orchestra.run.smithi008.stdout:Adding group ceph....done 2023-01-25T02:58:34.057 INFO:teuthology.orchestra.run.smithi008.stdout:Adding system user ceph....done 2023-01-25T02:58:34.226 INFO:teuthology.orchestra.run.smithi008.stdout:Setting system user ceph properties....done 2023-01-25T02:58:34.243 INFO:teuthology.orchestra.run.smithi008.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-01-25T02:58:34.394 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-01-25T02:58:34.718 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-01-25T02:58:35.205 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-test (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:35.348 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up radosgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:35.621 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-25T02:58:35.621 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-25T02:58:36.492 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-base (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:37.079 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-01-25T02:58:37.552 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mds (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:37.791 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-25T02:58:37.791 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-25T02:58:38.221 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:38.477 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-25T02:58:38.477 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-25T02:58:38.905 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-osd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:39.445 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-25T02:58:39.445 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-25T02:58:39.907 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-k8sevents (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:40.033 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:40.193 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mon (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:40.423 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-25T02:58:40.423 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-25T02:58:40.894 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:41.037 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:41.179 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-mgr-dashboard (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:41.322 INFO:teuthology.orchestra.run.smithi008.stdout:Setting up ceph-volume (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T02:58:41.457 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-25T02:58:41.659 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-01-25T02:58:42.169 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-25T02:58:45.247 INFO:teuthology.orchestra.run.smithi008.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-01-25T02:58:45.856 INFO:teuthology.orchestra.run.smithi008.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-25T02:58:45.859 DEBUG:teuthology.parallel:result is None 2023-01-25T02:58:45.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2023-01-25T02:58:45.905 INFO:teuthology.orchestra.run.smithi008.stdout:Ubuntu 2023-01-25T02:58:45.910 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-25T02:58:45.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T02:58:46.061 DEBUG:teuthology.orchestra.run.smithi008:> dpkg-query -W -f '${Version}' ceph 2023-01-25T02:58:46.086 INFO:teuthology.orchestra.run.smithi008.stdout:18.0.0-1963-g61996d73-1focal 2023-01-25T02:58:46.086 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1963-g61996d73-1focal 2023-01-25T02:58:46.087 INFO:teuthology.task.install:The correct ceph version 18.0.0-1963-g61996d73-1focal is installed. 2023-01-25T02:58:46.087 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-25T02:58:46.088 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:58:46.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-25T02:58:46.099 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-25T02:58:46.099 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:58:46.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/daemon-helper 2023-01-25T02:58:46.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-25T02:58:46.128 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-25T02:58:46.129 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:58:46.129 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-25T02:58:46.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-25T02:58:46.201 INFO:teuthology.run_tasks:Running task workunit... 2023-01-25T02:58:46.230 INFO:tasks.workunit:Pulling workunits from ref 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T02:58:46.231 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-25T02:58:46.231 INFO:tasks.workunit:timeout=3h 2023-01-25T02:58:46.232 INFO:tasks.workunit:cleanup=True 2023-01-25T02:58:46.232 DEBUG:teuthology.orchestra.run.smithi008:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-25T02:58:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T02:58:46.240 INFO:teuthology.orchestra.run.smithi008.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-25T02:58:46.240 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-25T02:58:46.287 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-25T02:58:46.288 DEBUG:teuthology.orchestra.run.smithi008:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-25T02:58:46.294 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T02:58:46.303 INFO:tasks.workunit.client.0.smithi008.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-25T02:59:28.586 INFO:tasks.workunit.client.0.smithi008.stderr:Note: switching to '61996d73ab738798e4ee0055c86457f5c459da4a'. 2023-01-25T02:59:28.587 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.587 INFO:tasks.workunit.client.0.smithi008.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-25T02:59:28.587 INFO:tasks.workunit.client.0.smithi008.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-25T02:59:28.587 INFO:tasks.workunit.client.0.smithi008.stderr:state without impacting any branches by switching back to a branch. 2023-01-25T02:59:28.587 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.588 INFO:tasks.workunit.client.0.smithi008.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-25T02:59:28.588 INFO:tasks.workunit.client.0.smithi008.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-25T02:59:28.588 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.588 INFO:tasks.workunit.client.0.smithi008.stderr: git switch -c 2023-01-25T02:59:28.588 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.589 INFO:tasks.workunit.client.0.smithi008.stderr:Or undo this operation with: 2023-01-25T02:59:28.589 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.589 INFO:tasks.workunit.client.0.smithi008.stderr: git switch - 2023-01-25T02:59:28.589 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.589 INFO:tasks.workunit.client.0.smithi008.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-25T02:59:28.590 INFO:tasks.workunit.client.0.smithi008.stderr: 2023-01-25T02:59:28.590 INFO:tasks.workunit.client.0.smithi008.stderr:HEAD is now at 61996d73ab7 Merge branch 'wip-tserong-qa-crash-chown' of https://github.com/SUSE/ceph into wip-yuri2-testing-2023-01-23-0928 2023-01-25T02:59:28.591 DEBUG:teuthology.orchestra.run.smithi008:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-25T02:59:28.598 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2023-01-25T02:59:28.599 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-25T02:59:28.647 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2023-01-25T02:59:28.648 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2023-01-25T02:59:28.648 DEBUG:teuthology.orchestra.run.smithi008:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61996d73ab738798e4ee0055c86457f5c459da4a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2023-01-25T02:59:28.687 INFO:tasks.workunit.client.0.smithi008.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-01-25T02:59:28.704 INFO:tasks.workunit.client.0.smithi008.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-01-25T02:59:28.704 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-25T02:59:28.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-25T02:59:28.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: export PYTHONWARNINGS=ignore 2023-01-25T02:59:28.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: PYTHONWARNINGS=ignore 2023-01-25T02:59:28.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export CEPH_CONF=/dev/null 2023-01-25T02:59:28.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: CEPH_CONF=/dev/null 2023-01-25T02:59:28.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: unset CEPH_ARGS 2023-01-25T02:59:28.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: local code 2023-01-25T02:59:28.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: main: run td/osd-mapper 2023-01-25T02:59:28.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2023-01-25T02:59:28.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2023-01-25T02:59:28.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2023-01-25T02:59:28.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2023-01-25T02:59:28.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2023-01-25T02:59:28.709 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2023-01-25T02:59:28.709 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none ' 2023-01-25T02:59:28.710 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2023-01-25T02:59:28.710 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-01-25T02:59:28.710 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2023-01-25T02:59:28.710 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-01-25T02:59:28.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2023-01-25T02:59:28.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2023-01-25T02:59:28.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2023-01-25T02:59:28.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2023-01-25T02:59:28.713 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2023-01-25T02:59:28.713 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-01-25T02:59:28.713 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T02:59:28.713 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-01-25T02:59:28.714 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T02:59:28.714 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T02:59:28.714 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T02:59:28.714 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T02:59:28.714 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T02:59:28.715 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T02:59:28.715 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T02:59:28.716 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T02:59:28.717 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T02:59:28.718 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T02:59:28.719 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T02:59:28.720 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T02:59:28.721 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T02:59:28.721 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T02:59:28.722 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T02:59:28.722 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T02:59:28.723 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T02:59:28.724 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T02:59:28.724 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-01-25T02:59:28.725 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T02:59:28.726 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T02:59:28.726 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T02:59:28.726 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22920 2023-01-25T02:59:28.727 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T02:59:28.727 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T02:59:28.728 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2023-01-25T02:59:28.729 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T02:59:28.730 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T02:59:28.730 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T02:59:28.730 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.22920 2023-01-25T02:59:28.732 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T02:59:28.733 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-01-25T02:59:28.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-01-25T02:59:28.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T02:59:28.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2023-01-25T02:59:28.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2023-01-25T02:59:28.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2023-01-25T02:59:28.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']="3" ['pgs_in_pool']="4" ['pool_name']="test") 2023-01-25T02:59:28.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2023-01-25T02:59:28.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2023-01-25T02:59:28.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2023-01-25T02:59:28.736 INFO:tasks.workunit.client.0.smithi008.stdout:Dir: td/osd-mapper 2023-01-25T02:59:28.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2023-01-25T02:59:28.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2023-01-25T02:59:28.738 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2023-01-25T02:59:28.738 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2023-01-25T02:59:28.738 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2023-01-25T02:59:28.738 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2023-01-25T02:59:28.738 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2023-01-25T02:59:28.739 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2023-01-25T02:59:28.739 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2023-01-25T02:59:28.739 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2023-01-25T02:59:28.739 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2023-01-25T02:59:28.739 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2023-01-25T02:59:30.702 INFO:tasks.workunit.client.0.smithi008.stdout:add osd0 e9250dbd-4e9b-4ad2-bb5e-4bb2bf8d149f 2023-01-25T02:59:31.002 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T02:59:31.152 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.146+0000 7ffa8bf2e680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T02:59:31.172 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.166+0000 7ffa8bf2e680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T02:59:31.175 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.170+0000 7ffa8bf2e680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T02:59:31.176 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.170+0000 7ffa8bf2e680 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-01-25T02:59:31.176 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.170+0000 7ffa8bf2e680 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-01-25T02:59:31.176 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.170+0000 7ffa8bf2e680 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-01-25T02:59:31.234 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:31.230+0000 7ffa8bf2e680 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2023-01-25T02:59:37.762 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd0 key to auth repository 2023-01-25T02:59:37.949 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.0 2023-01-25T02:59:37.983 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:37.978+0000 7f54a553f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T02:59:37.986 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:37.982+0000 7f54a553f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T02:59:37.987 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:37.982+0000 7f54a553f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T02:59:38.174 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T02:59:38.571 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:38.566+0000 7f54a553f680 -1 Falling back to public interface 2023-01-25T02:59:39.398 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T02:59:40.757 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T02:59:40.988 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T02:59:40.982+0000 7f54a553f680 -1 osd.0 0 log_to_monitors true 2023-01-25T02:59:42.096 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T02:59:43.412 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T02:59:44.855 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T02:59:46.179 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T02:59:47.493 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T02:59:48.891 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T02:59:50.209 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T02:59:51.666 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T02:59:53.210 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T02:59:54.644 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T02:59:56.155 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T02:59:57.780 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T02:59:59.096 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:00:00.667 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:00:02.731 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:00:04.171 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:00:05.798 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:00:07.209 INFO:tasks.workunit.client.0.smithi008.stdout:20 2023-01-25T03:00:07.516 INFO:tasks.workunit.client.0.smithi008.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/23957,v1:127.0.0.1:6801/23957] [v2:127.0.0.1:6802/23957,v1:127.0.0.1:6803/23957] exists,up e9250dbd-4e9b-4ad2-bb5e-4bb2bf8d149f 2023-01-25T03:00:07.536 INFO:tasks.workunit.client.0.smithi008.stdout:add osd1 9949d09b-c9e7-4402-8620-6f12d70616f0 2023-01-25T03:00:07.950 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:00:08.005 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:07.998+0000 7f26bb8ef680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:08.008 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:08.002+0000 7f26bb8ef680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:08.011 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:08.006+0000 7f26bb8ef680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:08.012 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:08.006+0000 7f26bb8ef680 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-01-25T03:00:08.012 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:08.006+0000 7f26bb8ef680 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-01-25T03:00:08.013 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:08.006+0000 7f26bb8ef680 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-01-25T03:00:08.075 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:08.070+0000 7f26bb8ef680 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2023-01-25T03:00:14.741 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd1 key to auth repository 2023-01-25T03:00:15.124 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.1 2023-01-25T03:00:15.160 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:15.154+0000 7f21d1fc1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:15.163 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:15.158+0000 7f21d1fc1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:15.164 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:15.158+0000 7f21d1fc1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:15.499 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:00:15.704 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:15.698+0000 7f21d1fc1680 -1 Falling back to public interface 2023-01-25T03:00:16.865 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:00:18.097 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:18.090+0000 7f21d1fc1680 -1 osd.1 0 log_to_monitors true 2023-01-25T03:00:18.179 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:00:19.547 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:00:20.862 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:00:22.184 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:00:23.496 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:00:24.844 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:00:26.174 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:00:27.560 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:00:29.150 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:00:30.462 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:00:31.999 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:00:33.481 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:00:35.020 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:00:36.725 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:00:38.165 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:00:39.920 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:00:41.309 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:00:41.644 INFO:tasks.workunit.client.0.smithi008.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25903,v1:127.0.0.1:6811/25903] [v2:127.0.0.1:6812/25903,v1:127.0.0.1:6813/25903] exists,up 9949d09b-c9e7-4402-8620-6f12d70616f0 2023-01-25T03:00:41.651 INFO:tasks.workunit.client.0.smithi008.stdout:add osd2 2bb2027a-aea8-4622-8a81-063144b96e3c 2023-01-25T03:00:42.064 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:00:42.122 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.119+0000 7fac8cd6f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:42.125 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.123+0000 7fac8cd6f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:42.127 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.123+0000 7fac8cd6f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:42.128 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.123+0000 7fac8cd6f680 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-01-25T03:00:42.128 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.123+0000 7fac8cd6f680 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-01-25T03:00:42.128 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.123+0000 7fac8cd6f680 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-01-25T03:00:42.204 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:42.199+0000 7fac8cd6f680 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2023-01-25T03:00:48.532 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd2 key to auth repository 2023-01-25T03:00:48.922 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.2 2023-01-25T03:00:48.957 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:48.951+0000 7f4eadf23680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:48.959 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:48.955+0000 7f4eadf23680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:48.960 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:48.955+0000 7f4eadf23680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:00:49.262 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:00:49.518 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:49.515+0000 7f4eadf23680 -1 Falling back to public interface 2023-01-25T03:00:50.609 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:00:51.921 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:00:51.946 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:00:51.943+0000 7f4eadf23680 -1 osd.2 0 log_to_monitors true 2023-01-25T03:00:53.237 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:00:54.549 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:00:55.925 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:00:57.317 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:00:58.738 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:01:00.052 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:01:01.371 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:01:02.776 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:01:04.095 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:01:05.785 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:01:07.103 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:01:08.699 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:01:10.232 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:01:12.062 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:01:13.680 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:01:15.055 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:01:15.433 INFO:tasks.workunit.client.0.smithi008.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/27667,v1:127.0.0.1:6819/27667] [v2:127.0.0.1:6820/27667,v1:127.0.0.1:6821/27667] exists,up 2bb2027a-aea8-4622-8a81-063144b96e3c 2023-01-25T03:01:15.950 INFO:tasks.workunit.client.0.smithi008.stderr:pool 'test' created 2023-01-25T03:01:18.129 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.0 seq 21474836502 2023-01-25T03:01:19.886 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.1 seq 42949672974 2023-01-25T03:01:20.254 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.2 seq 64424509449 2023-01-25T03:01:25.071 INFO:tasks.workunit.client.0.smithi008.stdout:standard_scrub_cluster: dbg: test pool is test 1 2023-01-25T03:01:25.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2023-01-25T03:01:25.203 INFO:tasks.workunit.client.0.smithi008.stdout:osd.0: { 2023-01-25T03:01:25.203 INFO:tasks.workunit.client.0.smithi008.stdout: "success": "osd_mclock_scheduler_background_best_effort_res = '10' osd_mclock_scheduler_background_recovery_lim = '41' osd_mclock_scheduler_background_recovery_res = '10' osd_mclock_scheduler_client_res = '20' osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T03:01:25.203 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:25.217 INFO:tasks.workunit.client.0.smithi008.stdout:osd.1: { 2023-01-25T03:01:25.218 INFO:tasks.workunit.client.0.smithi008.stdout: "success": "osd_mclock_scheduler_background_best_effort_res = '14' osd_mclock_scheduler_background_recovery_lim = '55' osd_mclock_scheduler_background_recovery_res = '14' osd_mclock_scheduler_client_res = '28' osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T03:01:25.218 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:25.233 INFO:tasks.workunit.client.0.smithi008.stdout:osd.2: { 2023-01-25T03:01:25.233 INFO:tasks.workunit.client.0.smithi008.stdout: "success": "osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T03:01:25.233 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:25.251 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2023-01-25T03:01:25.403 INFO:tasks.workunit.client.0.smithi008.stdout:osd.0: { 2023-01-25T03:01:25.403 INFO:tasks.workunit.client.0.smithi008.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T03:01:25.404 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:25.417 INFO:tasks.workunit.client.0.smithi008.stdout:osd.1: { 2023-01-25T03:01:25.418 INFO:tasks.workunit.client.0.smithi008.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T03:01:25.418 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:25.432 INFO:tasks.workunit.client.0.smithi008.stdout:osd.2: { 2023-01-25T03:01:25.432 INFO:tasks.workunit.client.0.smithi008.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-01-25T03:01:25.433 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:25.444 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2023-01-25T03:01:25.444 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2023-01-25T03:01:25.444 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2023-01-25T03:01:25.445 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2023-01-25T03:01:25.445 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2023-01-25T03:01:25.445 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-01-25T03:01:25.445 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-01-25T03:01:25.851 INFO:tasks.workunit.client.0.smithi008.stdout:created pool test snap snap01 2023-01-25T03:01:26.145 INFO:tasks.workunit.client.0.smithi008.stdout:created pool test snap snap02 2023-01-25T03:01:26.150 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2023-01-25T03:01:26.150 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-01-25T03:01:26.151 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-01-25T03:01:26.706 INFO:tasks.workunit.client.0.smithi008.stdout:created pool test snap snap13 2023-01-25T03:01:26.711 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2023-01-25T03:01:26.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-01-25T03:01:26.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-01-25T03:01:27.342 INFO:tasks.workunit.client.0.smithi008.stdout:created pool test snap snap24 2023-01-25T03:01:27.652 INFO:tasks.workunit.client.0.smithi008.stdout:created pool test snap snap25 2023-01-25T03:01:27.658 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 20307 2023-01-25T03:01:27.658 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2023-01-25T03:01:28.104 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-01-25T03:01:28.104 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2023-01-25T03:01:28.422 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2023-01-25T03:01:28.423 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-01-25T03:01:28.424 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2023-01-25T03:01:28.750 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2023-01-25T03:01:28.751 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2023-01-25T03:01:28.751 INFO:tasks.workunit.client.0.smithi008.stdout:pgid is 1.3 (primary: osd.1) 2023-01-25T03:01:28.751 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2023-01-25T03:01:28.752 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:293: set_query_debug: local pgid=1.3 2023-01-25T03:01:28.752 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: ceph pg dump pgs_brief 2023-01-25T03:01:28.753 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2023-01-25T03:01:29.038 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs_brief 2023-01-25T03:01:29.053 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local prim_osd=1 2023-01-25T03:01:29.053 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2023-01-25T03:01:29.053 INFO:tasks.workunit.client.0.smithi008.stdout:Setting scrub debug data. Primary for 1.3 is 1 2023-01-25T03:01:29.054 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: get_asok_path osd.1 2023-01-25T03:01:29.054 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-25T03:01:29.055 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-25T03:01:29.056 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T03:01:29.056 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:01:29.056 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T03:01:29.056 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.22920/ceph-osd.1.asok 2023-01-25T03:01:29.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: CEPH_ARGS= 2023-01-25T03:01:29.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.22920/ceph-osd.1.asok scrubdebug 1.3 set sessions 2023-01-25T03:01:29.162 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:29.162 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-01-25T03:01:29.205 INFO:tasks.workunit.client.0.smithi008.stdout:{"success":true}{ 2023-01-25T03:01:29.205 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "objxxx", 2023-01-25T03:01:29.205 INFO:tasks.workunit.client.0.smithi008.stdout: "clones": [ 2023-01-25T03:01:29.206 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.206 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 2, 2023-01-25T03:01:29.206 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [ 2023-01-25T03:01:29.206 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.206 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 1, 2023-01-25T03:01:29.207 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap01" 2023-01-25T03:01:29.207 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:01:29.207 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.207 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 2, 2023-01-25T03:01:29.207 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap02" 2023-01-25T03:01:29.208 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:01:29.208 INFO:tasks.workunit.client.0.smithi008.stdout: ], 2023-01-25T03:01:29.208 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6, 2023-01-25T03:01:29.208 INFO:tasks.workunit.client.0.smithi008.stdout: "overlaps": [] 2023-01-25T03:01:29.209 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:01:29.209 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.209 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 3, 2023-01-25T03:01:29.209 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [ 2023-01-25T03:01:29.210 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.210 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 3, 2023-01-25T03:01:29.210 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap13" 2023-01-25T03:01:29.210 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:01:29.211 INFO:tasks.workunit.client.0.smithi008.stdout: ], 2023-01-25T03:01:29.211 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6, 2023-01-25T03:01:29.211 INFO:tasks.workunit.client.0.smithi008.stdout: "overlaps": [] 2023-01-25T03:01:29.211 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:01:29.212 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.212 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 5, 2023-01-25T03:01:29.212 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [ 2023-01-25T03:01:29.212 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.212 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 4, 2023-01-25T03:01:29.213 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap24" 2023-01-25T03:01:29.213 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:01:29.213 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.213 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 5, 2023-01-25T03:01:29.214 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap25" 2023-01-25T03:01:29.214 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:01:29.214 INFO:tasks.workunit.client.0.smithi008.stdout: ], 2023-01-25T03:01:29.214 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6, 2023-01-25T03:01:29.214 INFO:tasks.workunit.client.0.smithi008.stdout: "overlaps": [] 2023-01-25T03:01:29.215 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:01:29.215 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:01:29.215 INFO:tasks.workunit.client.0.smithi008.stdout: "id": "head", 2023-01-25T03:01:29.216 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [], 2023-01-25T03:01:29.216 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6 2023-01-25T03:01:29.216 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:01:29.216 INFO:tasks.workunit.client.0.smithi008.stdout: ] 2023-01-25T03:01:29.216 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:29.217 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-01-25T03:01:29.338 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:01:29.339 INFO:tasks.workunit.client.0.smithi008.stdout: "deep": true, 2023-01-25T03:01:29.339 INFO:tasks.workunit.client.0.smithi008.stdout: "stamp": "2023-01-18T02:59:49.337653+0000" 2023-01-25T03:01:29.339 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:01:29.349 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2023-01-25T03:01:32.351 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T03:01:32.652 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:01:32.652 INFO:tasks.workunit.client.0.smithi008.stdout:1.3 2 0 0 0 0 12 0 0 3 0 3 active+clean 2023-01-25T03:01:17.239923+0000 25'3 25:19 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.652 INFO:tasks.workunit.client.0.smithi008.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:01:18.145432+0000 0'0 26:18 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.653 INFO:tasks.workunit.client.0.smithi008.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:01:16.770030+0000 0'0 28:19 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.653 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:01:17.240210+0000 0'0 25:17 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.653 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:32.653 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:01:32.655 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:01:32.664 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2023-01-25T03:01:32.668 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T03:01:32.990 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:01:32.991 INFO:tasks.workunit.client.0.smithi008.stdout:1.3 2 0 0 0 0 12 0 0 3 0 3 active+clean 2023-01-25T03:01:17.239923+0000 25'3 25:19 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.991 INFO:tasks.workunit.client.0.smithi008.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:01:18.145432+0000 0'0 26:18 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.991 INFO:tasks.workunit.client.0.smithi008.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:01:16.770030+0000 0'0 28:19 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.992 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:01:17.240210+0000 0'0 25:17 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:01:32.992 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:32.993 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:01:32.994 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:01:33.002 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2023-01-25T03:01:33.444 INFO:tasks.workunit.client.0.smithi008.stderr:noscrub is set 2023-01-25T03:01:33.456 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2023-01-25T03:01:33.956 INFO:tasks.workunit.client.0.smithi008.stderr:nodeep-scrub is set 2023-01-25T03:01:33.969 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2023-01-25T03:01:38.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2023-01-25T03:01:38.973 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-01-25T03:01:38.975 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:01:38.975 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:01:38.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:01:38.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:01:38.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:01:39.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:01:39.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:01:39.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-01-25T03:01:39.326 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bdev(0x555f758e6000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T03:01:39.326 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bdev(0x555f758e6000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:39.326 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T03:01:39.327 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bdev(0x555f758e6700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T03:01:39.327 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bdev(0x555f758e6700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:39.327 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T03:01:39.328 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.319+0000 7f32b4c31980 1 bluefs mount 2023-01-25T03:01:39.337 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.331+0000 7f32b4c31980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:39.369 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.363+0000 7f32b4c31980 1 bluefs mount shared_bdev_used = 0 2023-01-25T03:01:39.369 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.363+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:39.374 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:39.374 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:39.375 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:39.376 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:39.376 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:39.376 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:39.377 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:39.377 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:39.377 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.367+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:39.377 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.371+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:39.378 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.371+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:39.378 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.371+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:39.492 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.483+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:39.492 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.483+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2023-01-25T03:01:39.493 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.483+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-01-25T03:01:39.493 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.483+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T03:01:39.493 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.483+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-01-25T03:01:39.494 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.487+0000 7f32b4c31980 1 freelist init 2023-01-25T03:01:39.494 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.487+0000 7f32b4c31980 1 freelist _read_cfg 2023-01-25T03:01:39.495 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.491+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 1.14441e-07 2023-01-25T03:01:39.496 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.491+0000 7f32b4c31980 1 bluefs umount 2023-01-25T03:01:39.496 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.491+0000 7f32b4c31980 1 bdev(0x555f758e6700 td/osd-mapper/0/block) close 2023-01-25T03:01:39.608 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.603+0000 7f32b4c31980 1 bdev(0x555f758e6700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T03:01:39.608 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.603+0000 7f32b4c31980 1 bdev(0x555f758e6700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:39.609 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.603+0000 7f32b4c31980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T03:01:39.609 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.603+0000 7f32b4c31980 1 bluefs mount 2023-01-25T03:01:39.609 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.603+0000 7f32b4c31980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:39.612 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.607+0000 7f32b4c31980 1 bluefs mount shared_bdev_used = 20512768 2023-01-25T03:01:39.612 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.607+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:39.619 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:39.619 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:39.620 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:39.620 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:39.620 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:39.621 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:39.621 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:39.621 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:39.622 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:39.622 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:39.622 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:39.622 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:39.611+0000 7f32b4c31980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:40.581 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:40.575+0000 7f32b4c31980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:40.586 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:40.586 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:40.586 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:40.587 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:40.587 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T03:01:40.587 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:40.587 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:40.587 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.588 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:40.588 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T03:01:40.588 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:40.588 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:40.588 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T03:01:40.589 INFO:tasks.workunit.client.0.smithi008.stdout:0000003f 2023-01-25T03:01:40.589 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.589 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:40.589 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:40.589 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:40.590 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:40.590 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T03:01:40.590 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:40.590 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:40.590 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.591 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:40.591 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:40.591 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:40.591 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:40.591 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:40.592 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:40.592 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.592 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:40.592 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:40.592 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:40.593 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:40.593 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:40.593 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:40.593 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.593 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:40.594 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:40.594 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:40.594 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:40.594 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:40.594 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:40.595 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.595 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:40.595 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:40.595 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:40.595 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:40.595 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:40.596 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:40.596 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.596 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:40.596 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:40.597 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:40.597 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:40.597 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:40.597 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:40.597 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.597 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T03:01:40.598 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.598 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T03:01:40.598 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:40.598 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:40.579+0000 7f32b4c31980 1 bluefs umount 2023-01-25T03:01:40.599 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:40.579+0000 7f32b4c31980 1 bdev(0x555f758e6700 td/osd-mapper/0/block) close 2023-01-25T03:01:40.836 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:40.831+0000 7f32b4c31980 1 freelist shutdown 2023-01-25T03:01:40.836 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:40.831+0000 7f32b4c31980 1 bdev(0x555f758e6000 td/osd-mapper/0/block) close 2023-01-25T03:01:41.123 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:01:41.124 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T03:01:41.124 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2023-01-25T03:01:41.158 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bdev(0x5594fc204000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T03:01:41.159 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bdev(0x5594fc204000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:41.159 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T03:01:41.159 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bdev(0x5594fc204700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T03:01:41.159 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bdev(0x5594fc204700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:41.159 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T03:01:41.160 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.151+0000 7f47ab085980 1 bluefs mount 2023-01-25T03:01:41.179 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.171+0000 7f47ab085980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:41.204 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.199+0000 7f47ab085980 1 bluefs mount shared_bdev_used = 0 2023-01-25T03:01:41.204 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.199+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:41.210 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.203+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:41.211 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.203+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:41.211 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.203+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:41.211 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.203+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:41.212 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:41.212 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:41.213 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:41.214 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:41.214 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:41.214 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:41.214 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:41.215 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.207+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:41.334 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.327+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:41.335 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.327+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-01-25T03:01:41.335 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.327+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-01-25T03:01:41.335 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.327+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T03:01:41.335 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.327+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-01-25T03:01:41.336 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.331+0000 7f47ab085980 1 freelist init 2023-01-25T03:01:41.336 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.331+0000 7f47ab085980 1 freelist _read_cfg 2023-01-25T03:01:41.338 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.331+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 1.14441e-07 2023-01-25T03:01:41.339 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.331+0000 7f47ab085980 1 bluefs umount 2023-01-25T03:01:41.339 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.331+0000 7f47ab085980 1 bdev(0x5594fc204700 td/osd-mapper/2/block) close 2023-01-25T03:01:41.440 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bdev(0x5594fc204700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T03:01:41.440 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bdev(0x5594fc204700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:41.441 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T03:01:41.441 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bluefs mount 2023-01-25T03:01:41.441 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:41.443 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bluefs mount shared_bdev_used = 20447232 2023-01-25T03:01:41.443 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.435+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:41.449 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:41.450 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:41.450 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:41.450 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:41.450 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:41.450 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:41.451 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:41.451 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:41.452 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:41.452 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:41.452 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:41.452 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:41.443+0000 7f47ab085980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:42.386 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:42.383+0000 7f47ab085980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:42.391 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:42.383+0000 7f47ab085980 1 bluefs umount 2023-01-25T03:01:42.391 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:42.383+0000 7f47ab085980 1 bdev(0x5594fc204700 td/osd-mapper/2/block) close 2023-01-25T03:01:42.648 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:42.643+0000 7f47ab085980 1 freelist shutdown 2023-01-25T03:01:42.648 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:42.643+0000 7f47ab085980 1 bdev(0x5594fc204000 td/osd-mapper/2/block) close 2023-01-25T03:01:42.704 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:42.704 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:42.704 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:42.704 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:42.705 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:42.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:01:42.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2023-01-25T03:01:42.706 INFO:tasks.workunit.client.0.smithi008.stderr:grep: /tmp/oo2.dump: No such file or directory 2023-01-25T03:01:42.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:01:42.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T03:01:43.511 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:43.512 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:43.512 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:43.512 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:43.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T03:01:43.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:43.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:43.513 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.513 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:43.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T03:01:43.514 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:43.514 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:43.514 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T03:01:43.514 INFO:tasks.workunit.client.0.smithi008.stdout:0000003f 2023-01-25T03:01:43.515 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.515 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:43.515 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:43.516 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:43.516 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:43.516 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T03:01:43.516 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:43.517 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:43.517 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.518 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:43.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:43.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:43.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:43.519 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:43.519 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:43.519 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.519 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:43.519 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:43.520 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:43.520 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:43.520 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:43.520 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:43.520 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.521 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:43.521 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:43.521 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:43.521 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:43.521 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:43.522 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:43.522 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.522 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:43.522 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:43.522 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:43.523 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:43.523 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:43.523 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:43.523 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.523 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:43.524 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:43.524 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:43.524 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:43.524 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:43.524 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:43.524 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.525 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T03:01:43.525 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:43.525 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T03:01:43.525 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:44.031 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T03:01:44.031 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-01-25T03:01:44.032 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T03:01:44.032 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2023-01-25T03:01:45.608 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2023-01-25T03:01:45.609 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2023-01-25T03:01:45.611 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2023-01-25T03:01:45.612 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:01:45.613 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2023-01-25T03:01:45.613 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2023-01-25T03:01:45.613 INFO:tasks.workunit.client.0.smithi008.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2023-01-25T03:01:45.614 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-01-25T03:01:45.614 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-01-25T03:01:45.643 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bdev(0x55faf32c0000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T03:01:45.643 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bdev(0x55faf32c0000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:45.643 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T03:01:45.644 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bdev(0x55faf32c0700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T03:01:45.644 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bdev(0x55faf32c0700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:45.644 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T03:01:45.644 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.635+0000 7f768b915980 1 bluefs mount 2023-01-25T03:01:45.653 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.647+0000 7f768b915980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:45.660 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.655+0000 7f768b915980 1 bluefs mount shared_bdev_used = 0 2023-01-25T03:01:45.660 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.655+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:45.664 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:45.665 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:45.665 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:45.665 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:45.665 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:45.666 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:45.667 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:45.667 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:45.667 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:45.667 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:45.668 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:45.668 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.659+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:45.676 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:45.677 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2023-01-25T03:01:45.677 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-01-25T03:01:45.677 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T03:01:45.677 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-01-25T03:01:45.679 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 freelist init 2023-01-25T03:01:45.679 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.671+0000 7f768b915980 1 freelist _read_cfg 2023-01-25T03:01:45.683 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.679+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 1.14441e-07 2023-01-25T03:01:45.684 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.679+0000 7f768b915980 1 bluefs umount 2023-01-25T03:01:45.684 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.679+0000 7f768b915980 1 bdev(0x55faf32c0700 td/osd-mapper/0/block) close 2023-01-25T03:01:45.924 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.919+0000 7f768b915980 1 bdev(0x55faf32c0700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-01-25T03:01:45.924 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.919+0000 7f768b915980 1 bdev(0x55faf32c0700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:45.925 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.919+0000 7f768b915980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-01-25T03:01:45.925 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.919+0000 7f768b915980 1 bluefs mount 2023-01-25T03:01:45.925 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.919+0000 7f768b915980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:45.928 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.923+0000 7f768b915980 1 bluefs mount shared_bdev_used = 6881280 2023-01-25T03:01:45.928 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.923+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:45.934 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:45.934 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:45.934 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:45.935 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:45.935 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:45.936 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:45.936 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:45.936 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:45.936 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:45.936 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:45.937 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:45.937 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:45.927+0000 7f768b915980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:46.351 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:46.343+0000 7f768b915980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:46.355 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:46.356 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:46.356 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:46.356 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:46.356 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T03:01:46.356 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:46.357 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:46.357 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.357 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:46.357 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T03:01:46.357 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:46.358 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:46.358 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T03:01:46.358 INFO:tasks.workunit.client.0.smithi008.stdout:0000003f 2023-01-25T03:01:46.358 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.358 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:46.359 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:46.359 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:46.359 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:46.359 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T03:01:46.359 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:46.360 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:46.360 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.360 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:46.360 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:46.360 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:46.361 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:46.361 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:46.361 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:46.361 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.361 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:46.362 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:46.362 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:46.362 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:46.362 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:46.362 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:46.363 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.363 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:46.363 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:46.363 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:46.363 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:46.363 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:46.364 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:46.364 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.364 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:46.364 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:46.365 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:46.365 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:46.365 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:46.365 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:46.365 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.365 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:46.366 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:46.366 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:46.366 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:46.366 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:46.366 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:46.367 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.367 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T03:01:46.367 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.367 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T03:01:46.367 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:46.368 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:46.351+0000 7f768b915980 1 bluefs umount 2023-01-25T03:01:46.368 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:46.351+0000 7f768b915980 1 bdev(0x55faf32c0700 td/osd-mapper/0/block) close 2023-01-25T03:01:46.608 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:46.603+0000 7f768b915980 1 freelist shutdown 2023-01-25T03:01:46.608 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:46.603+0000 7f768b915980 1 bdev(0x55faf32c0000 td/osd-mapper/0/block) close 2023-01-25T03:01:46.679 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-01-25T03:01:46.679 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-01-25T03:01:46.679 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T03:01:48.011 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:48.011 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:48.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-01-25T03:01:48.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-01-25T03:01:48.013 INFO:tasks.workunit.client.0.smithi008.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:48.016 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-01-25T03:01:48.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.dFA6VTodmq_the_val 2023-01-25T03:01:48.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:48.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.dFA6VTodmq_the_val' 2023-01-25T03:01:48.017 INFO:tasks.workunit.client.0.smithi008.stdout:Value dumped in: /tmp/tmp.dFA6VTodmq_the_val 2023-01-25T03:01:48.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.dFA6VTodmq_the_val 2023-01-25T03:01:48.751 INFO:tasks.workunit.client.0.smithi008.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-01-25T03:01:49.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:01:49.094 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.dFA6VTodmq_the_val 2023-01-25T03:01:49.111 INFO:tasks.workunit.client.0.smithi008.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-01-25T03:01:49.111 INFO:tasks.workunit.client.0.smithi008.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-01-25T03:01:49.111 INFO:tasks.workunit.client.0.smithi008.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-01-25T03:01:49.111 INFO:tasks.workunit.client.0.smithi008.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-01-25T03:01:49.112 INFO:tasks.workunit.client.0.smithi008.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-01-25T03:01:49.112 INFO:tasks.workunit.client.0.smithi008.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2023-01-25T03:01:49.112 INFO:tasks.workunit.client.0.smithi008.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-01-25T03:01:49.112 INFO:tasks.workunit.client.0.smithi008.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-01-25T03:01:49.113 INFO:tasks.workunit.client.0.smithi008.stdout:0000073 2023-01-25T03:01:49.113 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T03:01:49.113 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:50.434 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.dFA6VTodmq_the_val 2023-01-25T03:01:51.770 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:51.770 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:01:51.770 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-01-25T03:01:51.771 INFO:tasks.workunit.client.0.smithi008.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2023-01-25T03:01:51.771 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2023-01-25T03:01:51.772 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-01-25T03:01:51.772 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T03:01:51.803 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bdev(0x55c24af34000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-01-25T03:01:51.803 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bdev(0x55c24af34000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:51.804 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T03:01:51.804 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bdev(0x55c24af34700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-01-25T03:01:51.804 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bdev(0x55c24af34700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:51.804 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-01-25T03:01:51.804 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.795+0000 7fbf593e0980 1 bluefs mount 2023-01-25T03:01:51.810 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.803+0000 7fbf593e0980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:51.836 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.831+0000 7fbf593e0980 1 bluefs mount shared_bdev_used = 0 2023-01-25T03:01:51.837 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.831+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:51.842 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:51.843 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:51.843 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:51.843 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:51.843 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:51.845 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:51.845 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:51.845 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.835+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:51.845 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.839+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:51.846 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.839+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:51.846 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.839+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:51.846 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.839+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:51.858 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.851+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:51.858 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.851+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2023-01-25T03:01:51.858 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.851+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2023-01-25T03:01:51.858 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.851+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T03:01:51.859 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.851+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2023-01-25T03:01:51.860 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.855+0000 7fbf593e0980 1 freelist init 2023-01-25T03:01:51.860 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.855+0000 7fbf593e0980 1 freelist _read_cfg 2023-01-25T03:01:51.862 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.855+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 1.14441e-07 2023-01-25T03:01:51.863 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.855+0000 7fbf593e0980 1 bluefs umount 2023-01-25T03:01:51.863 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:51.855+0000 7fbf593e0980 1 bdev(0x55c24af34700 td/osd-mapper/1/block) close 2023-01-25T03:01:52.092 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.087+0000 7fbf593e0980 1 bdev(0x55c24af34700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-01-25T03:01:52.093 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.087+0000 7fbf593e0980 1 bdev(0x55c24af34700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:52.093 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.087+0000 7fbf593e0980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-01-25T03:01:52.093 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.087+0000 7fbf593e0980 1 bluefs mount 2023-01-25T03:01:52.094 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.087+0000 7fbf593e0980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:52.096 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.091+0000 7fbf593e0980 1 bluefs mount shared_bdev_used = 6815744 2023-01-25T03:01:52.096 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.091+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:52.105 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:52.105 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:52.106 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:52.106 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:52.106 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:52.106 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:52.106 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:52.107 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:52.107 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:52.107 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:52.107 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:52.107 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.099+0000 7fbf593e0980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:52.505 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.499+0000 7fbf593e0980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:52.509 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:52.509 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:52.509 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:52.510 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:52.510 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T03:01:52.510 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:52.510 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:52.510 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.511 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:52.511 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T03:01:52.511 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:52.511 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:52.511 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T03:01:52.512 INFO:tasks.workunit.client.0.smithi008.stdout:0000003f 2023-01-25T03:01:52.512 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.512 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:52.512 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:52.512 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:52.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:52.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T03:01:52.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:52.513 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:52.513 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.513 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:52.514 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:52.514 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:52.514 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:52.515 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:52.516 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:52.516 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.516 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:52.516 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:52.516 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:52.517 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:52.517 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:52.517 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:52.517 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.517 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:52.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:52.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:52.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:52.518 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:52.518 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:52.519 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.519 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:52.519 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:52.520 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:52.520 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:52.520 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:52.520 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:52.520 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.521 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:52.521 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:52.521 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:52.522 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:52.522 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:52.522 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:52.522 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.523 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T03:01:52.523 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.523 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T03:01:52.523 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:52.524 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.503+0000 7fbf593e0980 1 bluefs umount 2023-01-25T03:01:52.524 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.503+0000 7fbf593e0980 1 bdev(0x55c24af34700 td/osd-mapper/1/block) close 2023-01-25T03:01:52.760 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.755+0000 7fbf593e0980 1 freelist shutdown 2023-01-25T03:01:52.760 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:52.755+0000 7fbf593e0980 1 bdev(0x55c24af34000 td/osd-mapper/1/block) close 2023-01-25T03:01:52.847 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T03:01:52.848 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-01-25T03:01:52.848 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T03:01:53.924 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:53.924 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:53.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-01-25T03:01:53.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-01-25T03:01:53.926 INFO:tasks.workunit.client.0.smithi008.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:53.926 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-01-25T03:01:53.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.VeXaT5M5Hk_the_val 2023-01-25T03:01:53.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:53.927 INFO:tasks.workunit.client.0.smithi008.stdout:Value dumped in: /tmp/tmp.VeXaT5M5Hk_the_val 2023-01-25T03:01:53.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.VeXaT5M5Hk_the_val' 2023-01-25T03:01:53.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.VeXaT5M5Hk_the_val 2023-01-25T03:01:54.676 INFO:tasks.workunit.client.0.smithi008.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-01-25T03:01:55.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:01:55.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.VeXaT5M5Hk_the_val 2023-01-25T03:01:55.010 INFO:tasks.workunit.client.0.smithi008.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-01-25T03:01:55.011 INFO:tasks.workunit.client.0.smithi008.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-01-25T03:01:55.011 INFO:tasks.workunit.client.0.smithi008.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-01-25T03:01:55.011 INFO:tasks.workunit.client.0.smithi008.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-01-25T03:01:55.011 INFO:tasks.workunit.client.0.smithi008.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-01-25T03:01:55.012 INFO:tasks.workunit.client.0.smithi008.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2023-01-25T03:01:55.012 INFO:tasks.workunit.client.0.smithi008.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-01-25T03:01:55.012 INFO:tasks.workunit.client.0.smithi008.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-01-25T03:01:55.012 INFO:tasks.workunit.client.0.smithi008.stdout:0000073 2023-01-25T03:01:55.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T03:01:55.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:56.085 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.VeXaT5M5Hk_the_val 2023-01-25T03:01:57.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:57.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:01:57.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2023-01-25T03:01:57.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2023-01-25T03:01:57.411 INFO:tasks.workunit.client.0.smithi008.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2023-01-25T03:01:57.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-01-25T03:01:57.412 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T03:01:57.445 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bdev(0x55d049608000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T03:01:57.445 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bdev(0x55d049608000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:57.446 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-25T03:01:57.446 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bdev(0x55d049608700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T03:01:57.446 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bdev(0x55d049608700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:57.446 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T03:01:57.446 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.439+0000 7fe81d239980 1 bluefs mount 2023-01-25T03:01:57.452 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.447+0000 7fe81d239980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:57.459 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.451+0000 7fe81d239980 1 bluefs mount shared_bdev_used = 0 2023-01-25T03:01:57.459 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.451+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:57.464 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.455+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:57.464 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:57.464 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:57.465 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:57.465 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:57.466 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:57.466 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:57.466 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:57.467 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:57.467 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:57.467 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:57.467 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.459+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:57.475 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.467+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:57.476 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.471+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-01-25T03:01:57.476 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.471+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-01-25T03:01:57.476 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.471+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-01-25T03:01:57.476 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.471+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-01-25T03:01:57.478 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.471+0000 7fe81d239980 1 freelist init 2023-01-25T03:01:57.478 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.471+0000 7fe81d239980 1 freelist _read_cfg 2023-01-25T03:01:57.483 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.475+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 4 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 1.14441e-07 2023-01-25T03:01:57.484 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.479+0000 7fe81d239980 1 bluefs umount 2023-01-25T03:01:57.484 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.479+0000 7fe81d239980 1 bdev(0x55d049608700 td/osd-mapper/2/block) close 2023-01-25T03:01:57.724 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.719+0000 7fe81d239980 1 bdev(0x55d049608700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-01-25T03:01:57.724 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.719+0000 7fe81d239980 1 bdev(0x55d049608700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-01-25T03:01:57.725 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.719+0000 7fe81d239980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-01-25T03:01:57.725 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.719+0000 7fe81d239980 1 bluefs mount 2023-01-25T03:01:57.735 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.727+0000 7fe81d239980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-01-25T03:01:57.739 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.731+0000 7fe81d239980 1 bluefs mount shared_bdev_used = 6815744 2023-01-25T03:01:57.739 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.731+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-01-25T03:01:57.746 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.739+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) 2023-01-25T03:01:57.747 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.739+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) 2023-01-25T03:01:57.747 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.739+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 2, name: m-1) 2023-01-25T03:01:57.748 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.739+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 3, name: m-2) 2023-01-25T03:01:57.748 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.739+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 4, name: p-0) 2023-01-25T03:01:57.748 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 5, name: p-1) 2023-01-25T03:01:57.748 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 6, name: p-2) 2023-01-25T03:01:57.749 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 7, name: O-0) 2023-01-25T03:01:57.749 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 8, name: O-1) 2023-01-25T03:01:57.749 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 9, name: O-2) 2023-01-25T03:01:57.750 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 10, name: L) 2023-01-25T03:01:57.750 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:57.743+0000 7fe81d239980 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) 2023-01-25T03:01:58.233 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:58.231+0000 7fe81d239980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600 2023-01-25T03:01:58.237 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:58.237 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:58.237 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:58.237 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:58.238 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-01-25T03:01:58.238 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:58.238 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:58.238 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.238 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:58.239 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-01-25T03:01:58.239 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:58.239 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:58.239 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-01-25T03:01:58.239 INFO:tasks.workunit.client.0.smithi008.stdout:0000003f 2023-01-25T03:01:58.240 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.240 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:58.243 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-01-25T03:01:58.243 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-01-25T03:01:58.243 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-01-25T03:01:58.243 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-01-25T03:01:58.244 INFO:tasks.workunit.client.0.smithi008.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-01-25T03:01:58.244 INFO:tasks.workunit.client.0.smithi008.stdout:00000047 2023-01-25T03:01:58.244 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.244 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:58.244 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:58.245 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:58.245 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:58.245 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:58.245 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:58.245 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.246 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:01:58.246 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:58.246 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:58.246 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:58.247 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:58.247 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:58.247 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.247 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:58.247 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:58.248 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:58.248 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:58.248 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:58.248 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:58.249 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.249 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:58.249 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:58.249 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:58.249 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:58.250 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:58.250 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:58.250 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.250 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:01:58.251 INFO:tasks.workunit.client.0.smithi008.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-01-25T03:01:58.251 INFO:tasks.workunit.client.0.smithi008.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-01-25T03:01:58.251 INFO:tasks.workunit.client.0.smithi008.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-01-25T03:01:58.251 INFO:tasks.workunit.client.0.smithi008.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-01-25T03:01:58.251 INFO:tasks.workunit.client.0.smithi008.stdout:0000003b 2023-01-25T03:01:58.251 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.252 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-01-25T03:01:58.252 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.252 INFO:tasks.workunit.client.0.smithi008.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-01-25T03:01:58.252 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:01:58.253 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:58.231+0000 7fe81d239980 1 bluefs umount 2023-01-25T03:01:58.253 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:58.231+0000 7fe81d239980 1 bdev(0x55d049608700 td/osd-mapper/2/block) close 2023-01-25T03:01:58.496 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:58.491+0000 7fe81d239980 1 freelist shutdown 2023-01-25T03:01:58.496 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:01:58.491+0000 7fe81d239980 1 bdev(0x55d049608000 td/osd-mapper/2/block) close 2023-01-25T03:01:58.743 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-01-25T03:01:58.743 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-01-25T03:01:58.743 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T03:01:59.818 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:59.818 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:59.819 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-01-25T03:01:59.819 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-01-25T03:01:59.820 INFO:tasks.workunit.client.0.smithi008.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:01:59.821 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-01-25T03:01:59.822 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.yIFkONBwb8_the_val 2023-01-25T03:01:59.823 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:01:59.823 INFO:tasks.workunit.client.0.smithi008.stdout:Value dumped in: /tmp/tmp.yIFkONBwb8_the_val 2023-01-25T03:01:59.823 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.yIFkONBwb8_the_val' 2023-01-25T03:01:59.824 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.yIFkONBwb8_the_val 2023-01-25T03:02:00.549 INFO:tasks.workunit.client.0.smithi008.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-01-25T03:02:00.897 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:02:01.007 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.yIFkONBwb8_the_val 2023-01-25T03:02:01.008 INFO:tasks.workunit.client.0.smithi008.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-01-25T03:02:01.008 INFO:tasks.workunit.client.0.smithi008.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-01-25T03:02:01.008 INFO:tasks.workunit.client.0.smithi008.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-01-25T03:02:01.008 INFO:tasks.workunit.client.0.smithi008.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-01-25T03:02:01.008 INFO:tasks.workunit.client.0.smithi008.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-01-25T03:02:01.009 INFO:tasks.workunit.client.0.smithi008.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2023-01-25T03:02:01.009 INFO:tasks.workunit.client.0.smithi008.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-01-25T03:02:01.009 INFO:tasks.workunit.client.0.smithi008.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-01-25T03:02:01.009 INFO:tasks.workunit.client.0.smithi008.stdout:0000073 2023-01-25T03:02:01.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T03:02:01.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:02:01.981 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.yIFkONBwb8_the_val 2023-01-25T03:02:03.305 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:02:03.306 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off ' 2023-01-25T03:02:03.306 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off 2023-01-25T03:02:03.307 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:03.307 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:02:03.307 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/' 2023-01-25T03:02:03.308 INFO:tasks.workunit.client.0.smithi008.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off/ 2023-01-25T03:02:03.308 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2023-01-25T03:02:03.309 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2023-01-25T03:02:03.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:02:03.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:03.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2023-01-25T03:02:03.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-mapper 2023-01-25T03:02:03.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2023-01-25T03:02:03.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2023-01-25T03:02:03.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2023-01-25T03:02:03.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-mapper/0 2023-01-25T03:02:03.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:03.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:02:03.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:02:03.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:02:03.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2023-01-25T03:02:03.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2023-01-25T03:02:03.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2023-01-25T03:02:03.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2023-01-25T03:02:03.316 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-01-25T03:02:03.316 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2023-01-25T03:02:03.316 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:02:03.316 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:02:03.318 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:02:03.318 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:03.318 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T03:02:03.318 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22920/$cluster-$name.asok' 2023-01-25T03:02:03.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22920/$cluster-$name.asok' 2023-01-25T03:02:03.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:02:03.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-01-25T03:02:03.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-01-25T03:02:03.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:02:03.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:02:03.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:03.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2023-01-25T03:02:03.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2023-01-25T03:02:03.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-mapper/0 2023-01-25T03:02:03.321 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.0 2023-01-25T03:02:03.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2023-01-25T03:02:03.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.22920/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:03.322 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-mapper/0/whoami 2023-01-25T03:02:03.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2023-01-25T03:02:03.323 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2023-01-25T03:02:03.323 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2023-01-25T03:02:03.323 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2023-01-25T03:02:03.359 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:03.351+0000 7f44f9ac3680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:03.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:03.355+0000 7f44f9ac3680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:03.364 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:03.363+0000 7f44f9ac3680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:03.642 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2023-01-25T03:02:03.642 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:02:03.642 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T03:02:03.643 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:02:03.643 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:02:03.643 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:02:03.644 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:03.644 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:02:03.644 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:03.644 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:03.901 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:03.899+0000 7f44f9ac3680 -1 Falling back to public interface 2023-01-25T03:02:03.994 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:04.996 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:04.997 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:04.997 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:02:04.998 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:02:04.998 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:04.998 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:05.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:06.325 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:06.325 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:06.325 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:02:06.326 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:02:06.326 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:06.326 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:06.507 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:06.507+0000 7f44f9ac3680 -1 osd.0 30 log_to_monitors true 2023-01-25T03:02:06.650 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:07.652 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:07.652 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:07.653 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:02:07.653 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:02:07.654 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:07.654 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:08.008 INFO:tasks.workunit.client.0.smithi008.stdout:osd.0 up in weight 1 up_from 33 up_thru 33 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6800/36735,v1:127.0.0.1:6801/36735] [v2:127.0.0.1:6802/36735,v1:127.0.0.1:6803/36735] exists,up e9250dbd-4e9b-4ad2-bb5e-4bb2bf8d149f 2023-01-25T03:02:08.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:02:08.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:02:08.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:02:08.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:02:08.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:08.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2023-01-25T03:02:08.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-mapper 2023-01-25T03:02:08.013 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2023-01-25T03:02:08.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2023-01-25T03:02:08.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2023-01-25T03:02:08.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-mapper/1 2023-01-25T03:02:08.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:08.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:02:08.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:02:08.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:02:08.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2023-01-25T03:02:08.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2023-01-25T03:02:08.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2023-01-25T03:02:08.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2023-01-25T03:02:08.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-01-25T03:02:08.016 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2023-01-25T03:02:08.016 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:02:08.016 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:02:08.016 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:02:08.017 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:08.017 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T03:02:08.017 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22920/$cluster-$name.asok' 2023-01-25T03:02:08.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22920/$cluster-$name.asok' 2023-01-25T03:02:08.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:02:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-01-25T03:02:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-01-25T03:02:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:02:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:02:08.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:08.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2023-01-25T03:02:08.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2023-01-25T03:02:08.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-mapper/1 2023-01-25T03:02:08.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2023-01-25T03:02:08.020 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.1 2023-01-25T03:02:08.021 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.22920/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:08.021 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-mapper/1/whoami 2023-01-25T03:02:08.021 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2023-01-25T03:02:08.021 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2023-01-25T03:02:08.021 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2023-01-25T03:02:08.022 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2023-01-25T03:02:08.053 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:08.047+0000 7ffb665eb680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:08.073 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:08.071+0000 7ffb665eb680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:08.075 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:08.071+0000 7ffb665eb680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:08.353 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2023-01-25T03:02:08.353 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:02:08.353 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T03:02:08.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:02:08.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:02:08.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:08.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:02:08.355 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:02:08.355 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:08.355 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:02:08.614 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:08.611+0000 7ffb665eb680 -1 Falling back to public interface 2023-01-25T03:02:08.665 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:09.667 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:09.668 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:09.668 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:02:09.668 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:02:09.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:09.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:02:10.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:11.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:11.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:11.017 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:02:11.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:02:11.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:11.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:02:11.034 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:11.031+0000 7ffb665eb680 -1 osd.1 30 log_to_monitors true 2023-01-25T03:02:11.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:12.356 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:12.356 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:12.357 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:02:12.357 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:02:12.357 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:12.358 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:02:12.683 INFO:tasks.workunit.client.0.smithi008.stdout:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/37304,v1:127.0.0.1:6811/37304] [v2:127.0.0.1:6812/37304,v1:127.0.0.1:6813/37304] exists,up 9949d09b-c9e7-4402-8620-6f12d70616f0 2023-01-25T03:02:12.686 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:02:12.686 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:02:12.686 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:02:12.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:02:12.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:12.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2023-01-25T03:02:12.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-mapper 2023-01-25T03:02:12.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2023-01-25T03:02:12.688 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2023-01-25T03:02:12.688 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2023-01-25T03:02:12.688 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-mapper/2 2023-01-25T03:02:12.688 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2023-01-25T03:02:12.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:02:12.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:02:12.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:02:12.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2023-01-25T03:02:12.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2023-01-25T03:02:12.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2023-01-25T03:02:12.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2023-01-25T03:02:12.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-01-25T03:02:12.690 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2023-01-25T03:02:12.690 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:02:12.691 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:02:12.691 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:02:12.692 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:12.693 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T03:02:12.693 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22920/$cluster-$name.asok' 2023-01-25T03:02:12.693 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22920/$cluster-$name.asok' 2023-01-25T03:02:12.693 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:02:12.693 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-01-25T03:02:12.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-01-25T03:02:12.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:02:12.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:02:12.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:12.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2023-01-25T03:02:12.695 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2023-01-25T03:02:12.695 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-mapper/2 2023-01-25T03:02:12.695 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2023-01-25T03:02:12.695 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.2 2023-01-25T03:02:12.696 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=06fdc38c-5a85-4475-855d-248dfdfa6f33 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.22920/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:12.697 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-mapper/2/whoami 2023-01-25T03:02:12.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2023-01-25T03:02:12.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2023-01-25T03:02:12.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2023-01-25T03:02:12.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2023-01-25T03:02:12.730 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:12.727+0000 7f965350a680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:12.752 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:12.751+0000 7f965350a680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:12.754 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:12.751+0000 7f965350a680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:13.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2023-01-25T03:02:13.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:02:13.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T03:02:13.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:02:13.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:02:13.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:13.037 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:02:13.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:02:13.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:13.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:02:13.302 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:13.299+0000 7f965350a680 -1 Falling back to public interface 2023-01-25T03:02:13.408 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:14.409 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:14.409 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:14.410 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:02:14.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:02:14.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:14.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:02:14.743 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:15.456 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:15.455+0000 7f965350a680 -1 osd.2 30 log_to_monitors true 2023-01-25T03:02:15.745 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:15.745 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:15.745 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:02:15.746 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:02:15.746 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:15.746 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:02:16.081 INFO:tasks.workunit.client.0.smithi008.stdout:osd.2 up in weight 1 up_from 39 up_thru 17 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/37873,v1:127.0.0.1:6819/37873] [v2:127.0.0.1:6820/37873,v1:127.0.0.1:6821/37873] exists,up 2bb2027a-aea8-4622-8a81-063144b96e3c 2023-01-25T03:02:16.082 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:02:16.082 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:02:16.082 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:02:16.082 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2023-01-25T03:02:17.085 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2023-01-25T03:02:17.086 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2023-01-25T03:02:17.088 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:02:17.088 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2023-01-25T03:02:17.263 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:02:17.263 INFO:tasks.workunit.client.0.smithi008.stdout: "version": "18.0.0-1963-g61996d73", 2023-01-25T03:02:17.264 INFO:tasks.workunit.client.0.smithi008.stdout: "release": "reef", 2023-01-25T03:02:17.264 INFO:tasks.workunit.client.0.smithi008.stdout: "release_type": "dev" 2023-01-25T03:02:17.264 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:02:17.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:02:17.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2023-01-25T03:02:17.413 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:02:17.414 INFO:tasks.workunit.client.0.smithi008.stdout: "version": "18.0.0-1963-g61996d73", 2023-01-25T03:02:17.414 INFO:tasks.workunit.client.0.smithi008.stdout: "release": "reef", 2023-01-25T03:02:17.414 INFO:tasks.workunit.client.0.smithi008.stdout: "release_type": "dev" 2023-01-25T03:02:17.414 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:02:17.425 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-01-25T03:02:17.425 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2023-01-25T03:02:17.607 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:02:17.607 INFO:tasks.workunit.client.0.smithi008.stdout: "version": "18.0.0-1963-g61996d73", 2023-01-25T03:02:17.607 INFO:tasks.workunit.client.0.smithi008.stdout: "release": "reef", 2023-01-25T03:02:17.608 INFO:tasks.workunit.client.0.smithi008.stdout: "release_type": "dev" 2023-01-25T03:02:17.608 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:02:17.618 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-01-25T03:02:17.664 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:02:17.664 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "objxxx", 2023-01-25T03:02:17.664 INFO:tasks.workunit.client.0.smithi008.stdout: "clones": [ 2023-01-25T03:02:17.664 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.665 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 2, 2023-01-25T03:02:17.665 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [ 2023-01-25T03:02:17.665 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.665 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 1, 2023-01-25T03:02:17.665 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap01" 2023-01-25T03:02:17.666 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:02:17.666 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.666 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 2, 2023-01-25T03:02:17.666 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap02" 2023-01-25T03:02:17.666 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:02:17.667 INFO:tasks.workunit.client.0.smithi008.stdout: ], 2023-01-25T03:02:17.667 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6, 2023-01-25T03:02:17.667 INFO:tasks.workunit.client.0.smithi008.stdout: "overlaps": [] 2023-01-25T03:02:17.667 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:02:17.667 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.668 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 3, 2023-01-25T03:02:17.668 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [ 2023-01-25T03:02:17.668 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.668 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 3, 2023-01-25T03:02:17.668 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap13" 2023-01-25T03:02:17.669 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:02:17.669 INFO:tasks.workunit.client.0.smithi008.stdout: ], 2023-01-25T03:02:17.669 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6, 2023-01-25T03:02:17.669 INFO:tasks.workunit.client.0.smithi008.stdout: "overlaps": [] 2023-01-25T03:02:17.669 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:02:17.670 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.670 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 5, 2023-01-25T03:02:17.670 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [ 2023-01-25T03:02:17.670 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.670 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 4, 2023-01-25T03:02:17.670 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap24" 2023-01-25T03:02:17.671 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:02:17.671 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.671 INFO:tasks.workunit.client.0.smithi008.stdout: "id": 5, 2023-01-25T03:02:17.671 INFO:tasks.workunit.client.0.smithi008.stdout: "name": "snap25" 2023-01-25T03:02:17.671 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:02:17.672 INFO:tasks.workunit.client.0.smithi008.stdout: ], 2023-01-25T03:02:17.672 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6, 2023-01-25T03:02:17.672 INFO:tasks.workunit.client.0.smithi008.stdout: "overlaps": [] 2023-01-25T03:02:17.672 INFO:tasks.workunit.client.0.smithi008.stdout: }, 2023-01-25T03:02:17.672 INFO:tasks.workunit.client.0.smithi008.stdout: { 2023-01-25T03:02:17.673 INFO:tasks.workunit.client.0.smithi008.stdout: "id": "head", 2023-01-25T03:02:17.673 INFO:tasks.workunit.client.0.smithi008.stdout: "snapshots": [], 2023-01-25T03:02:17.673 INFO:tasks.workunit.client.0.smithi008.stdout: "size": 6 2023-01-25T03:02:17.673 INFO:tasks.workunit.client.0.smithi008.stdout: } 2023-01-25T03:02:17.673 INFO:tasks.workunit.client.0.smithi008.stdout: ] 2023-01-25T03:02:17.673 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:02:17.674 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2023-01-25T03:02:18.145 INFO:tasks.workunit.client.0.smithi008.stderr:nodeep-scrub is unset 2023-01-25T03:02:18.159 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2023-01-25T03:02:18.641 INFO:tasks.workunit.client.0.smithi008.stderr:noscrub is unset 2023-01-25T03:02:18.655 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2023-01-25T03:02:18.656 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-01-25T03:02:19.002 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2023-01-25T03:02:19.002 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T03:02:19.309 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:02:19.309 INFO:tasks.workunit.client.0.smithi008.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 peering 2023-01-25T03:02:16.174636+0000 28'7 39:54 [1,2,0] 1 [1,2,0] 1 28'7 2023-01-25T03:01:29.934989+0000 28'7 2023-01-25T03:01:29.934989+0000 0 1 periodic scrub scheduled @ 2023-01-26T03:01:29.934989+0000 4 0 2023-01-25T03:02:19.309 INFO:tasks.workunit.client.0.smithi008.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:16.782051+0000 0'0 40:47 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:19.310 INFO:tasks.workunit.client.0.smithi008.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2023-01-25T03:02:11.606206+0000 0'0 36:15 [0,1] 0 [0,1] 0 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:19.310 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2023-01-25T03:02:16.174753+0000 0'0 39:26 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:19.310 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:02:19.310 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:02:19.314 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:02:19.326 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2023-01-25T03:02:21.328 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-01-25T03:02:21.465 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:02:21.466 INFO:tasks.workunit.client.0.smithi008.stdout: "deep": true, 2023-01-25T03:02:21.466 INFO:tasks.workunit.client.0.smithi008.stdout: "stamp": "2023-01-18T03:00:41.464766+0000" 2023-01-25T03:02:21.466 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:02:21.476 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2023-01-25T03:02:26.478 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T03:02:26.781 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:02:26.781 INFO:tasks.workunit.client.0.smithi008.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2023-01-25T03:02:17.268995+0000 28'7 41:66 [1,2,0] 1 [1,2,0] 1 28'7 2023-01-25T03:01:29.934989+0000 28'7 2023-01-25T03:01:29.934989+0000 0 1 periodic scrub scheduled @ 2023-01-26T03:01:29.934989+0000 4 0 2023-01-25T03:02:26.782 INFO:tasks.workunit.client.0.smithi008.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:16.782051+0000 0'0 42:51 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:26.782 INFO:tasks.workunit.client.0.smithi008.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:16.681694+0000 0'0 41:27 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:26.782 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:17.192778+0000 0'0 41:37 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:26.782 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:02:26.782 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:02:26.785 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:02:26.794 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:02:26.795 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T03:02:26.798 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:02:22.139+0000 7ffb4640d700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-01-25T03:02:26.798 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2023-01-25T03:02:26.802 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T03:02:26.802 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2023-01-25T03:02:26.806 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2023-01-25T03:02:26.806 INFO:tasks.workunit.client.0.smithi008.stdout:prev count: 1 2023-01-25T03:02:26.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2023-01-25T03:02:26.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-01-25T03:02:26.933 INFO:tasks.workunit.client.0.smithi008.stdout:{ 2023-01-25T03:02:26.933 INFO:tasks.workunit.client.0.smithi008.stdout: "deep": true, 2023-01-25T03:02:26.933 INFO:tasks.workunit.client.0.smithi008.stdout: "stamp": "2023-01-18T03:00:46.932445+0000" 2023-01-25T03:02:26.934 INFO:tasks.workunit.client.0.smithi008.stdout:} 2023-01-25T03:02:26.943 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2023-01-25T03:02:31.945 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2023-01-25T03:02:32.244 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:02:32.244 INFO:tasks.workunit.client.0.smithi008.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2023-01-25T03:02:22.142439+0000 28'7 43:84 [1,2,0] 1 [1,2,0] 1 28'7 2023-01-25T03:02:22.142268+0000 28'7 2023-01-25T03:02:22.142268+0000 0 1 periodic scrub scheduled @ 2023-01-26T03:02:22.142268+0000 4 0 2023-01-25T03:02:32.244 INFO:tasks.workunit.client.0.smithi008.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:16.782051+0000 0'0 42:51 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:32.245 INFO:tasks.workunit.client.0.smithi008.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:16.681694+0000 0'0 41:27 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:32.245 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:02:17.192778+0000 0'0 41:37 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T03:01:15.858524+0000 0'0 2023-01-25T03:01:15.858524+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:01:15.858524+0000 0 0 2023-01-25T03:02:32.245 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:02:32.245 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:02:32.247 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:02:32.256 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:02:32.257 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T03:02:32.260 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:02:22.139+0000 7ffb4640d700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-01-25T03:02:32.261 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2023-01-25T03:02:32.261 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-01-25T03:02:32.265 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2023-01-25T03:02:32.266 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-01-25T03:02:32.266 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2023-01-25T03:02:32.266 INFO:tasks.workunit.client.0.smithi008.stdout:current count: 1 2023-01-25T03:02:32.267 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2023-01-25T03:02:32.267 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-01-25T03:02:32.267 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:02:32.268 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:02:32.268 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:02:32.268 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:02:32.268 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:02:32.579 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:02:32.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-01-25T03:02:32.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-01-25T03:02:32.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T03:02:32.581 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2023-01-25T03:02:32.581 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T03:02:34.150 INFO:tasks.workunit.client.0.smithi008.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:02:34.150 INFO:tasks.workunit.client.0.smithi008.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-01-25T03:02:34.150 INFO:tasks.workunit.client.0.smithi008.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-01-25T03:02:34.151 INFO:tasks.workunit.client.0.smithi008.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-01-25T03:02:34.151 INFO:tasks.workunit.client.0.smithi008.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:02:34.151 INFO:tasks.workunit.client.0.smithi008.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-01-25T03:02:34.152 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-01-25T03:02:34.152 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-01-25T03:02:34.153 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-01-25T03:02:34.153 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2023-01-25T03:02:35.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2023-01-25T03:02:35.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2023-01-25T03:02:35.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2023-01-25T03:02:35.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2023-01-25T03:02:35.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-01-25T03:02:35.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T03:02:35.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-01-25T03:02:35.229 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:02:35.229 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:02:35.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:02:35.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:02:35.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:02:35.345 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:02:35.345 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T03:02:35.347 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T03:02:35.348 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T03:02:35.350 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T03:02:35.350 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T03:02:35.351 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T03:02:35.353 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:02:35.353 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T03:02:35.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T03:02:35.354 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:02:35.355 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T03:02:35.358 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T03:02:35.358 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-01-25T03:02:35.380 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T03:02:35.380 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:35.380 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T03:02:35.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22920 2023-01-25T03:02:35.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T03:02:35.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T03:02:35.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: code=0 2023-01-25T03:02:35.383 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: main: teardown td/osd-mapper 0 2023-01-25T03:02:35.383 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-01-25T03:02:35.383 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-01-25T03:02:35.383 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-01-25T03:02:35.384 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:02:35.384 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:02:35.385 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:02:35.385 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:02:35.385 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:02:35.388 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:02:35.388 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T03:02:35.389 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T03:02:35.390 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T03:02:35.392 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T03:02:35.392 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T03:02:35.393 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T03:02:35.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:02:35.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T03:02:35.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T03:02:35.396 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:02:35.397 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T03:02:35.399 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-01-25T03:02:35.399 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-01-25T03:02:35.401 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T03:02:35.401 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:35.401 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22920 2023-01-25T03:02:35.402 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22920 2023-01-25T03:02:35.402 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T03:02:35.403 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T03:02:35.403 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: main: return 0 2023-01-25T03:02:35.404 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-25T03:02:35.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-25T03:02:35.421 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2023-01-25T03:02:35.421 DEBUG:teuthology.orchestra.run.smithi008:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61996d73ab738798e4ee0055c86457f5c459da4a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2023-01-25T03:02:35.482 INFO:tasks.workunit.client.0.smithi008.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-01-25T03:02:35.495 INFO:tasks.workunit.client.0.smithi008.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-01-25T03:02:35.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-25T03:02:35.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-25T03:02:35.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: export PYTHONWARNINGS=ignore 2023-01-25T03:02:35.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: PYTHONWARNINGS=ignore 2023-01-25T03:02:35.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export CEPH_CONF=/dev/null 2023-01-25T03:02:35.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: CEPH_CONF=/dev/null 2023-01-25T03:02:35.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: unset CEPH_ARGS 2023-01-25T03:02:35.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: local code 2023-01-25T03:02:35.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: main: run td/osd-recovery-scrub 2023-01-25T03:02:35.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2023-01-25T03:02:35.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2023-01-25T03:02:35.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2023-01-25T03:02:35.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2023-01-25T03:02:35.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2023-01-25T03:02:35.498 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2023-01-25T03:02:35.499 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none ' 2023-01-25T03:02:35.499 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2023-01-25T03:02:35.499 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:28: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-01-25T03:02:35.499 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: set 2023-01-25T03:02:35.500 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-01-25T03:02:35.503 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: local 'funcs=TEST_recovery_scrub_1 2023-01-25T03:02:35.503 INFO:tasks.workunit.client.0.smithi008.stderr:TEST_recovery_scrub_2' 2023-01-25T03:02:35.503 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2023-01-25T03:02:35.503 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2023-01-25T03:02:35.504 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:38: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2023-01-25T03:02:35.504 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local poolname=test 2023-01-25T03:02:35.504 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:41: TEST_recovery_scrub_1: TESTDATA=testdata.39517 2023-01-25T03:02:35.504 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: OSDS=4 2023-01-25T03:02:35.504 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: PGS=1 2023-01-25T03:02:35.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: OBJECTS=100 2023-01-25T03:02:35.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: ERRORS=0 2023-01-25T03:02:35.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2023-01-25T03:02:35.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-01-25T03:02:35.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-01-25T03:02:35.506 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-01-25T03:02:35.506 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T03:02:35.506 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-01-25T03:02:35.506 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:02:35.506 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:02:35.507 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:02:35.507 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:02:35.508 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:02:35.508 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:02:35.508 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T03:02:35.509 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T03:02:35.509 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T03:02:35.511 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T03:02:35.511 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T03:02:35.511 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T03:02:35.514 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:02:35.514 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T03:02:35.514 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T03:02:35.514 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:02:35.515 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T03:02:35.517 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T03:02:35.517 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-01-25T03:02:35.519 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T03:02:35.519 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:35.520 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:35.520 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.39517 2023-01-25T03:02:35.521 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T03:02:35.521 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T03:02:35.521 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-01-25T03:02:35.523 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T03:02:35.523 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:35.523 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:35.523 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.39517 2023-01-25T03:02:35.525 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T03:02:35.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-01-25T03:02:35.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-01-25T03:02:35.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T03:02:35.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-01-25T03:02:35.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T03:02:35.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-01-25T03:02:35.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-25T03:02:35.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-25T03:02:35.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-25T03:02:35.529 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-01-25T03:02:35.529 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T03:02:35.907 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-25T03:02:35.908 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:02:35.908 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:02:35.909 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:02:35.910 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:35.910 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:35.910 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:02:35.910 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T03:02:36.198 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-25T03:02:36.199 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-25T03:02:36.200 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T03:02:36.200 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T03:02:36.200 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-25T03:02:36.201 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-25T03:02:36.201 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T03:02:36.201 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T03:02:36.202 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T03:02:36.202 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T03:02:36.203 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:36.203 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:36.203 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.39517/ceph-mon.a.asok 2023-01-25T03:02:36.203 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T03:02:36.203 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.39517/ceph-mon.a.asok config get fsid 2023-01-25T03:02:36.299 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-25T03:02:36.299 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T03:02:36.300 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T03:02:36.300 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-25T03:02:36.301 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T03:02:36.302 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T03:02:36.302 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-25T03:02:36.302 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T03:02:36.302 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T03:02:36.302 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:36.303 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:36.303 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.39517/ceph-mon.a.asok 2023-01-25T03:02:36.303 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T03:02:36.303 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.39517/ceph-mon.a.asok config get mon_host 2023-01-25T03:02:36.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2023-01-25T03:02:36.396 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-01-25T03:02:36.396 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-25T03:02:36.396 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-25T03:02:36.396 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-25T03:02:36.397 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-01-25T03:02:36.397 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-25T03:02:37.263 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-25T03:02:37.264 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:02:37.264 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:02:37.265 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:02:37.265 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:37.265 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:37.265 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:02:37.266 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T03:02:37.266 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T03:02:37.308 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: expr 4 - 1 2023-01-25T03:02:37.310 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: seq 0 3 2023-01-25T03:02:37.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:02:37.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2023-01-25T03:02:37.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:02:37.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:02:37.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-25T03:02:37.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:02:37.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-01-25T03:02:37.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:02:37.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:02:37.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:02:37.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:02:37.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-01-25T03:02:37.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-01-25T03:02:37.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:02:37.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:02:37.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:02:37.318 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:02:37.319 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:02:37.319 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:02:37.319 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:02:37.319 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:02:37.319 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:02:37.320 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:02:37.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:02:37.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:02:37.320 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:02:37.321 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:02:37.321 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:02:37.321 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:02:37.321 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:02:37.321 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:02:37.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:02:37.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:02:37.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T03:02:37.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-01-25T03:02:37.323 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:02:37.323 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bcf7dd21-5e2c-4a40-9d4e-6302f28c6554 2023-01-25T03:02:37.323 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 bcf7dd21-5e2c-4a40-9d4e-6302f28c6554' 2023-01-25T03:02:37.324 INFO:tasks.workunit.client.0.smithi008.stdout:add osd0 bcf7dd21-5e2c-4a40-9d4e-6302f28c6554 2023-01-25T03:02:37.324 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:02:37.339 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDNm9Bjt0IaFBAAhg/Olk4QMqH3fkxQDuFjMQ== 2023-01-25T03:02:37.339 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDNm9Bjt0IaFBAAhg/Olk4QMqH3fkxQDuFjMQ=="}' 2023-01-25T03:02:37.339 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bcf7dd21-5e2c-4a40-9d4e-6302f28c6554 -i td/osd-recovery-scrub/0/new.json 2023-01-25T03:02:37.626 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:02:37.640 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-01-25T03:02:37.641 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQDNm9Bjt0IaFBAAhg/Olk4QMqH3fkxQDuFjMQ== --osd-uuid bcf7dd21-5e2c-4a40-9d4e-6302f28c6554 2023-01-25T03:02:37.675 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.671+0000 7f06b9110680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:37.716 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.715+0000 7f06b9110680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:37.718 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.715+0000 7f06b9110680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:37.719 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.719+0000 7f06b9110680 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T03:02:37.720 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.719+0000 7f06b9110680 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T03:02:37.720 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.719+0000 7f06b9110680 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T03:02:37.763 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:37.759+0000 7f06b9110680 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-01-25T03:02:43.844 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-01-25T03:02:43.844 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:02:43.846 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd0 key to auth repository 2023-01-25T03:02:43.846 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-25T03:02:43.846 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:02:44.229 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.0 2023-01-25T03:02:44.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-25T03:02:44.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T03:02:44.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:02:44.231 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:02:44.231 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:02:44.263 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:44.258+0000 7f09e11c5680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:44.266 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:44.262+0000 7f09e11c5680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:44.268 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:44.266+0000 7f09e11c5680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:02:44.530 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-25T03:02:44.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:02:44.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T03:02:44.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:02:44.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:02:44.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:44.532 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:02:44.532 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:02:44.533 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:44.533 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:44.826 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:44.822+0000 7f09e11c5680 -1 Falling back to public interface 2023-01-25T03:02:44.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:45.955 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:45.955 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:45.955 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:02:45.956 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:02:45.956 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:45.956 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:46.273 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:47.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:47.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:47.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:02:47.275 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:02:47.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:47.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:47.352 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:02:47.350+0000 7f09e11c5680 -1 osd.0 0 log_to_monitors true 2023-01-25T03:02:47.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:48.613 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:48.614 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:48.614 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:02:48.615 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:02:48.615 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:48.615 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:48.914 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:49.916 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:49.917 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:49.917 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:02:49.918 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:02:49.918 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:49.918 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:50.221 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:51.223 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:51.223 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:51.224 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:02:51.224 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:02:51.224 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:51.225 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:51.536 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:52.538 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:52.538 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:52.538 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:02:52.539 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:02:52.539 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:52.539 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:52.908 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:53.910 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:53.911 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:53.911 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:02:53.912 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:02:53.912 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:53.912 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:54.258 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:55.260 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:55.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:55.261 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:02:55.262 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:02:55.262 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:55.262 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:55.579 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:56.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:56.581 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:56.581 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:02:56.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:02:56.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:56.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:56.972 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:57.973 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:57.974 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:57.974 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:02:57.975 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:02:57.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:57.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:58.279 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:02:59.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:02:59.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:02:59.281 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:02:59.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:02:59.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:02:59.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:02:59.653 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:00.655 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:00.656 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:00.656 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:03:00.657 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:03:00.657 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:00.657 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:01.219 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:02.220 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:02.221 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:02.221 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:03:02.222 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:03:02.222 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:02.222 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:02.852 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:03.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:03.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:03.855 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:03:03.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:03:03.856 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:03.856 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:04.602 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:05.603 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:05.603 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:05.604 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:03:05.604 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:03:05.605 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:05.605 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:06.026 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:07.028 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:07.028 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:07.029 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:03:07.029 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:03:07.030 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:07.030 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:07.447 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:08.450 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:08.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:08.452 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:03:08.454 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:03:08.454 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:08.454 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:08.948 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:09.950 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:09.950 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:09.950 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:03:09.951 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:03:09.951 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:09.951 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:03:10.270 INFO:tasks.workunit.client.0.smithi008.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/40618,v1:127.0.0.1:6803/40618] [v2:127.0.0.1:6804/40618,v1:127.0.0.1:6805/40618] exists,up bcf7dd21-5e2c-4a40-9d4e-6302f28c6554 2023-01-25T03:03:10.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:03:10.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:03:10.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:03:10.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:03:10.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2023-01-25T03:03:10.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:03:10.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:03:10.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-25T03:03:10.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:03:10.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-01-25T03:03:10.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:03:10.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:03:10.276 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:03:10.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:03:10.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-01-25T03:03:10.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-01-25T03:03:10.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:03:10.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:03:10.278 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:03:10.278 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:03:10.278 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:03:10.278 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:03:10.279 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:03:10.279 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:03:10.279 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:03:10.279 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:03:10.280 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:03:10.280 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:03:10.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:03:10.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:03:10.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:03:10.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:03:10.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:03:10.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:03:10.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:03:10.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:03:10.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T03:03:10.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-01-25T03:03:10.283 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:03:10.283 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=ad4fc899-337d-412d-b09a-7d042b627d56 2023-01-25T03:03:10.284 INFO:tasks.workunit.client.0.smithi008.stdout:add osd1 ad4fc899-337d-412d-b09a-7d042b627d56 2023-01-25T03:03:10.284 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 ad4fc899-337d-412d-b09a-7d042b627d56' 2023-01-25T03:03:10.285 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:03:10.300 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDum9BjcS7CERAA7wV5UbMPsvX2+mTy0K8WfQ== 2023-01-25T03:03:10.300 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDum9BjcS7CERAA7wV5UbMPsvX2+mTy0K8WfQ=="}' 2023-01-25T03:03:10.300 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new ad4fc899-337d-412d-b09a-7d042b627d56 -i td/osd-recovery-scrub/1/new.json 2023-01-25T03:03:10.788 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:03:10.804 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-01-25T03:03:10.806 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQDum9BjcS7CERAA7wV5UbMPsvX2+mTy0K8WfQ== --osd-uuid ad4fc899-337d-412d-b09a-7d042b627d56 2023-01-25T03:03:10.843 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.838+0000 7fc3e9788680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:10.847 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.842+0000 7fc3e9788680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:10.849 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.846+0000 7fc3e9788680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:10.849 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.846+0000 7fc3e9788680 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T03:03:10.850 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.846+0000 7fc3e9788680 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T03:03:10.850 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.846+0000 7fc3e9788680 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T03:03:10.970 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:10.966+0000 7fc3e9788680 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-01-25T03:03:17.240 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-01-25T03:03:17.241 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:03:17.242 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd1 key to auth repository 2023-01-25T03:03:17.242 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-25T03:03:17.243 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:03:17.632 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-25T03:03:17.632 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.1 2023-01-25T03:03:17.633 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:03:17.634 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T03:03:17.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:03:17.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:03:17.667 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:17.662+0000 7ffa9c5c7680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:17.670 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:17.666+0000 7ffa9c5c7680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:17.671 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:17.666+0000 7ffa9c5c7680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:17.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-25T03:03:17.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:03:17.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T03:03:17.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:03:17.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:03:17.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:17.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:03:17.935 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:03:17.935 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:17.935 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:18.227 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:18.222+0000 7ffa9c5c7680 -1 Falling back to public interface 2023-01-25T03:03:18.260 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:19.262 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:19.263 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:19.263 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:03:19.264 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:03:19.264 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:19.264 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:19.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:20.581 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:20.574+0000 7ffa9c5c7680 -1 osd.1 0 log_to_monitors true 2023-01-25T03:03:20.608 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:20.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:20.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:03:20.609 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:03:20.610 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:20.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:21.066 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:22.067 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:22.068 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:22.068 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:03:22.069 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:03:22.069 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:22.069 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:22.412 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:23.413 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:23.413 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:23.414 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:03:23.414 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:03:23.415 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:23.415 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:23.732 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:24.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:24.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:24.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:03:24.735 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:03:24.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:24.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:25.043 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:26.045 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:26.046 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:26.046 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:03:26.046 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:03:26.047 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:26.047 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:26.407 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:27.408 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:27.409 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:27.409 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:03:27.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:03:27.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:27.410 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:27.851 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:28.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:28.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:28.856 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:03:28.857 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:03:28.857 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:28.857 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:29.193 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:30.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:30.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:30.195 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:03:30.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:03:30.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:30.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:30.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:31.533 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:31.533 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:31.534 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:03:31.534 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:03:31.535 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:31.535 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:31.938 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:32.940 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:32.941 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:32.941 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:03:32.941 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:03:32.942 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:32.942 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:33.672 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:34.673 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:34.674 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:34.674 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:03:34.676 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:03:34.676 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:34.676 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:35.310 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:36.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:36.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:36.313 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:03:36.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:03:36.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:36.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:36.679 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:37.680 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:37.681 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:37.681 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:03:37.682 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:03:37.683 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:37.683 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:38.003 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:39.006 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:39.006 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:39.007 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:03:39.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:03:39.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:39.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:39.449 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:40.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:40.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:40.452 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:03:40.452 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:03:40.453 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:40.453 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:41.343 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:42.345 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:42.345 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:42.345 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:03:42.346 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:03:42.346 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:42.346 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:42.747 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:43.749 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:43.750 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:43.750 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:03:43.751 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:03:43.751 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:43.751 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:03:44.061 INFO:tasks.workunit.client.0.smithi008.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/42381,v1:127.0.0.1:6811/42381] [v2:127.0.0.1:6812/42381,v1:127.0.0.1:6813/42381] exists,up ad4fc899-337d-412d-b09a-7d042b627d56 2023-01-25T03:03:44.063 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:03:44.063 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:03:44.063 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:03:44.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:03:44.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2023-01-25T03:03:44.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:03:44.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:03:44.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-25T03:03:44.065 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:03:44.065 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-01-25T03:03:44.065 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:03:44.065 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:03:44.065 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:03:44.066 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:03:44.066 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-01-25T03:03:44.066 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-01-25T03:03:44.066 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:03:44.066 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:03:44.067 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:03:44.067 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:03:44.067 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:03:44.067 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:03:44.069 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:03:44.070 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:03:44.070 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:03:44.070 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:03:44.070 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:03:44.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:03:44.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:03:44.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:03:44.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:03:44.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:03:44.072 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:03:44.072 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:03:44.072 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:03:44.072 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:03:44.072 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T03:03:44.073 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-01-25T03:03:44.073 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:03:44.073 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=c67c2ab5-6603-47b3-b29a-f78ad6a13501 2023-01-25T03:03:44.073 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 c67c2ab5-6603-47b3-b29a-f78ad6a13501' 2023-01-25T03:03:44.074 INFO:tasks.workunit.client.0.smithi008.stdout:add osd2 c67c2ab5-6603-47b3-b29a-f78ad6a13501 2023-01-25T03:03:44.074 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:03:44.095 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAQnNBjN/SPBRAAFNGfFSC3OUNXM/y97g+qFg== 2023-01-25T03:03:44.095 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAQnNBjN/SPBRAAFNGfFSC3OUNXM/y97g+qFg=="}' 2023-01-25T03:03:44.095 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new c67c2ab5-6603-47b3-b29a-f78ad6a13501 -i td/osd-recovery-scrub/2/new.json 2023-01-25T03:03:44.548 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:03:44.566 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-01-25T03:03:44.568 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQAQnNBjN/SPBRAAFNGfFSC3OUNXM/y97g+qFg== --osd-uuid c67c2ab5-6603-47b3-b29a-f78ad6a13501 2023-01-25T03:03:44.605 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.601+0000 7f33d3a39680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:44.608 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.605+0000 7f33d3a39680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:44.610 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.605+0000 7f33d3a39680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:44.611 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.605+0000 7f33d3a39680 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T03:03:44.612 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.605+0000 7f33d3a39680 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T03:03:44.612 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.605+0000 7f33d3a39680 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T03:03:44.681 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:44.677+0000 7f33d3a39680 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-01-25T03:03:51.108 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-01-25T03:03:51.108 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:03:51.110 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd2 key to auth repository 2023-01-25T03:03:51.111 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-25T03:03:51.111 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:03:51.522 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.2 2023-01-25T03:03:51.524 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-25T03:03:51.524 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T03:03:51.524 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:03:51.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:03:51.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:03:51.555 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:51.549+0000 7f01e28d8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:51.557 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:51.553+0000 7f01e28d8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:51.560 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:51.553+0000 7f01e28d8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:03:51.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-25T03:03:51.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:03:51.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T03:03:51.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:03:51.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:03:51.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:51.868 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:03:51.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:03:51.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:51.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:03:52.093 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:52.089+0000 7f01e28d8680 -1 Falling back to public interface 2023-01-25T03:03:52.242 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:53.243 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:53.244 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:53.244 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:03:53.245 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:03:53.245 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:53.245 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:03:53.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:54.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:54.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:54.564 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:03:54.564 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:03:54.565 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:54.565 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:03:54.632 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:03:54.625+0000 7f01e28d8680 -1 osd.2 0 log_to_monitors true 2023-01-25T03:03:54.859 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:55.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:55.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:55.861 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:03:55.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:03:55.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:55.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:03:56.166 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:57.168 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:57.168 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:57.169 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:03:57.169 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:03:57.170 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:57.170 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:03:57.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:58.584 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:58.584 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:58.584 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:03:58.585 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:03:58.586 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:58.586 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:03:58.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:03:59.897 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:03:59.898 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:03:59.898 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:03:59.899 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:03:59.899 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:03:59.899 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:00.257 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:01.259 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:01.259 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:01.260 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:04:01.260 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:04:01.260 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:01.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:01.688 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:02.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:02.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:02.690 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:04:02.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:04:02.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:02.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:03.034 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:04.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:04.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:04.036 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:04:04.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:04:04.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:04.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:04.456 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:05.458 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:05.458 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:05.458 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:04:05.459 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:04:05.459 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:05.459 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:05.903 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:06.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:06.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:06.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:04:06.906 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:04:06.906 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:06.906 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:07.311 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:08.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:08.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:08.313 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:04:08.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:04:08.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:08.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:08.917 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:09.919 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:09.919 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:09.920 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:04:09.920 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:04:09.920 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:09.921 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:10.224 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:11.226 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:11.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:11.227 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:04:11.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:04:11.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:11.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:11.770 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:12.772 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:12.773 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:12.773 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:04:12.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:04:12.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:12.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:13.330 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:14.332 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:14.333 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:14.333 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:04:14.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:04:14.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:14.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:14.886 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:15.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:15.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:15.889 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:04:15.889 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:04:15.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:15.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:16.600 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:17.602 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:17.603 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:17.603 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:04:17.604 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:04:17.604 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:17.604 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:04:18.051 INFO:tasks.workunit.client.0.smithi008.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44154,v1:127.0.0.1:6819/44154] [v2:127.0.0.1:6820/44154,v1:127.0.0.1:6821/44154] exists,up c67c2ab5-6603-47b3-b29a-f78ad6a13501 2023-01-25T03:04:18.055 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:04:18.055 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:04:18.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:04:18.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:04:18.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2023-01-25T03:04:18.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:04:18.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:04:18.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-01-25T03:04:18.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:04:18.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-01-25T03:04:18.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:04:18.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:04:18.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:04:18.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:04:18.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-01-25T03:04:18.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-01-25T03:04:18.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:04:18.059 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:04:18.059 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:04:18.059 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:04:18.059 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:04:18.059 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:04:18.060 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:04:18.060 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:04:18.060 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:04:18.060 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:04:18.060 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:04:18.061 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:04:18.061 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:04:18.061 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:04:18.061 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:04:18.061 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:04:18.062 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:04:18.062 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:04:18.062 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:04:18.062 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:04:18.062 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-01-25T03:04:18.063 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-01-25T03:04:18.064 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:04:18.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0bd02485-36ea-4f20-9f3a-d140d2ade1b3 2023-01-25T03:04:18.064 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 0bd02485-36ea-4f20-9f3a-d140d2ade1b3' 2023-01-25T03:04:18.065 INFO:tasks.workunit.client.0.smithi008.stdout:add osd3 0bd02485-36ea-4f20-9f3a-d140d2ade1b3 2023-01-25T03:04:18.065 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:04:18.079 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAynNBjWxOYBBAAeeYy6+iRwm8NrYhNoR5pCA== 2023-01-25T03:04:18.079 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAynNBjWxOYBBAAeeYy6+iRwm8NrYhNoR5pCA=="}' 2023-01-25T03:04:18.080 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0bd02485-36ea-4f20-9f3a-d140d2ade1b3 -i td/osd-recovery-scrub/3/new.json 2023-01-25T03:04:18.537 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:04:18.555 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-01-25T03:04:18.557 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --mkfs --key AQAynNBjWxOYBBAAeeYy6+iRwm8NrYhNoR5pCA== --osd-uuid 0bd02485-36ea-4f20-9f3a-d140d2ade1b3 2023-01-25T03:04:18.594 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.589+0000 7f62ff037680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:04:18.610 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.605+0000 7f62ff037680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:04:18.613 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.609+0000 7f62ff037680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:04:18.614 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.609+0000 7f62ff037680 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T03:04:18.614 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.609+0000 7f62ff037680 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T03:04:18.614 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.609+0000 7f62ff037680 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T03:04:18.680 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:18.673+0000 7f62ff037680 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-01-25T03:04:25.097 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-01-25T03:04:25.097 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:04:25.098 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd3 key to auth repository 2023-01-25T03:04:25.098 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2023-01-25T03:04:25.098 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:04:25.526 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.3 2023-01-25T03:04:25.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2023-01-25T03:04:25.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false 2023-01-25T03:04:25.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:04:25.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:04:25.528 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:04:25.556 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:25.549+0000 7f246b3a8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:04:25.562 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:25.553+0000 7f246b3a8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:04:25.563 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:25.557+0000 7f246b3a8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:04:25.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2023-01-25T03:04:25.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:04:25.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=3 2023-01-25T03:04:25.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:04:25.872 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:04:25.872 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:25.872 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:04:25.872 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:04:25.873 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:25.873 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:26.106 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:26.101+0000 7f246b3a8680 -1 Falling back to public interface 2023-01-25T03:04:26.203 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:27.205 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:27.205 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:27.205 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:04:27.206 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:04:27.206 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:27.207 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:27.522 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:28.524 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:28.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:28.525 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:04:28.526 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:04:28.526 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:28.526 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:28.742 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:04:28.737+0000 7f246b3a8680 -1 osd.3 0 log_to_monitors true 2023-01-25T03:04:28.839 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:29.841 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:29.841 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:29.841 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:04:29.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:04:29.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:29.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:30.164 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:31.167 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:31.167 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:31.168 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:04:31.168 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:04:31.169 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:31.169 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:31.633 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:32.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:32.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:32.635 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:04:32.636 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:04:32.636 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:32.637 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:32.999 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:34.001 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:34.001 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:34.001 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:04:34.002 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:04:34.002 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:34.002 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:34.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:35.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:35.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:35.318 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:04:35.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:04:35.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:35.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:35.632 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:36.634 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:36.634 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:36.634 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:04:36.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:04:36.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:36.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:37.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:38.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:38.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:38.037 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:04:38.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:04:38.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:38.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:38.471 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:39.473 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:39.473 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:39.473 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:04:39.474 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:04:39.474 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:39.475 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:39.898 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:40.899 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:40.900 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:40.900 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:04:40.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:04:40.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:40.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:41.271 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:42.273 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:42.273 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:42.274 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:04:42.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:04:42.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:42.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:42.859 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:43.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:43.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:43.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:04:43.862 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:04:43.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:43.863 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:44.628 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:45.630 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:45.630 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:45.631 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:04:45.631 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:04:45.632 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:45.632 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:46.115 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:47.116 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:47.117 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:47.117 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:04:47.117 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:04:47.118 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:47.118 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:47.608 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:48.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:48.610 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:48.610 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:04:48.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:04:48.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:48.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:49.414 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:50.417 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:50.417 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:50.417 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:04:50.418 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:04:50.418 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:50.418 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:50.977 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:04:51.979 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:04:51.979 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:04:51.980 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:04:51.980 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:04:51.981 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:04:51.981 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:04:52.291 INFO:tasks.workunit.client.0.smithi008.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/45917,v1:127.0.0.1:6827/45917] [v2:127.0.0.1:6828/45917,v1:127.0.0.1:6829/45917] exists,up 0bd02485-36ea-4f20-9f3a-d140d2ade1b3 2023-01-25T03:04:52.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:04:52.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:04:52.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:04:52.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: create_pool test 1 1 2023-01-25T03:04:52.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-01-25T03:04:52.777 INFO:tasks.workunit.client.0.smithi008.stderr:pool 'test' created 2023-01-25T03:04:52.790 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T03:04:53.793 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub_1: wait_for_clean 2023-01-25T03:04:53.793 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T03:04:53.793 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T03:04:53.793 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T03:04:53.794 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T03:04:53.794 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T03:04:53.795 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T03:04:53.795 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T03:04:53.795 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T03:04:53.796 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T03:04:53.796 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T03:04:54.005 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T03:04:54.005 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T03:04:54.005 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T03:04:54.005 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T03:04:54.006 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T03:04:54.326 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T03:04:54.326 INFO:tasks.workunit.client.0.smithi008.stderr:1 2023-01-25T03:04:54.326 INFO:tasks.workunit.client.0.smithi008.stderr:2 2023-01-25T03:04:54.326 INFO:tasks.workunit.client.0.smithi008.stderr:3' 2023-01-25T03:04:54.327 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T03:04:54.327 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:04:54.327 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T03:04:54.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836505 2023-01-25T03:04:54.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836505 2023-01-25T03:04:54.499 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505' 2023-01-25T03:04:54.499 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:04:54.499 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T03:04:54.639 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672983 2023-01-25T03:04:54.639 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672983 2023-01-25T03:04:54.640 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2023-01-25T03:04:54.640 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:04:54.640 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T03:04:54.809 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509455 2023-01-25T03:04:54.810 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509455 2023-01-25T03:04:54.810 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509455' 2023-01-25T03:04:54.810 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:04:54.811 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T03:04:54.955 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345933 2023-01-25T03:04:54.956 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345933 2023-01-25T03:04:54.956 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509455 3-85899345933' 2023-01-25T03:04:54.956 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:04:54.957 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836505 2023-01-25T03:04:54.957 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:04:54.958 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T03:04:54.959 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836505 2023-01-25T03:04:54.959 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:04:54.961 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836505 2023-01-25T03:04:54.961 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2023-01-25T03:04:54.962 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.0 seq 21474836505 2023-01-25T03:04:54.962 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T03:04:55.260 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836505 -lt 21474836505 2023-01-25T03:04:55.260 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:04:55.261 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672983 2023-01-25T03:04:55.261 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:04:55.263 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T03:04:55.264 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672983 2023-01-25T03:04:55.265 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:04:55.267 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672983 2023-01-25T03:04:55.267 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2023-01-25T03:04:55.267 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.1 seq 42949672983 2023-01-25T03:04:55.268 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T03:04:55.594 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672983 -lt 42949672983 2023-01-25T03:04:55.595 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:04:55.596 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509455 2023-01-25T03:04:55.596 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:04:55.597 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T03:04:55.599 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509455 2023-01-25T03:04:55.599 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:04:55.600 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509455 2023-01-25T03:04:55.600 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.2 seq 64424509455 2023-01-25T03:04:55.601 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2023-01-25T03:04:55.601 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T03:04:55.899 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509454 -lt 64424509455 2023-01-25T03:04:55.900 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-25T03:04:56.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-25T03:04:56.902 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T03:04:57.213 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509455 -lt 64424509455 2023-01-25T03:04:57.214 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:04:57.215 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345933 2023-01-25T03:04:57.215 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:04:57.217 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T03:04:57.218 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345933 2023-01-25T03:04:57.218 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:04:57.220 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345933 2023-01-25T03:04:57.220 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2023-01-25T03:04:57.220 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.3 seq 85899345933 2023-01-25T03:04:57.221 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T03:04:57.583 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345933 -lt 85899345933 2023-01-25T03:04:57.584 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T03:04:57.585 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T03:04:57.586 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T03:04:57.988 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 1 == 0 2023-01-25T03:04:57.988 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T03:04:57.989 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T03:04:57.989 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T03:04:57.989 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T03:04:57.990 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T03:04:57.990 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T03:04:57.990 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T03:04:58.339 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=1 2023-01-25T03:04:58.340 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T03:04:58.341 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T03:04:58.341 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T03:04:58.713 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 1 = 1 2023-01-25T03:04:58.713 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T03:04:58.714 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T03:04:58.714 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: ceph osd dump 2023-01-25T03:04:58.715 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2023-01-25T03:04:58.715 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: awk '{ print $2 }' 2023-01-25T03:04:59.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: poolid=1 2023-01-25T03:04:59.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: ceph pg dump pgs 2023-01-25T03:04:59.342 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:04:59.342 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T03:04:53.562619+0000 0'0 22:8 [1] 1 [1] 1 0'0 2023-01-25T03:04:52.638763+0000 0'0 2023-01-25T03:04:52.638763+0000 0 0 periodic scrub scheduled @ 2023-01-26T08:57:15.165455+0000 0 0 2023-01-25T03:04:59.342 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:04:59.342 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:04:59.345 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:04:59.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.39517 bs=1M count=50 2023-01-25T03:04:59.509 INFO:tasks.workunit.client.0.smithi008.stderr:50+0 records in 2023-01-25T03:04:59.510 INFO:tasks.workunit.client.0.smithi008.stderr:50+0 records out 2023-01-25T03:04:59.510 INFO:tasks.workunit.client.0.smithi008.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.153213 s, 342 MB/s 2023-01-25T03:04:59.511 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: seq 1 100 2023-01-25T03:04:59.512 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:04:59.512 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj1 testdata.39517 2023-01-25T03:05:01.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:01.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj2 testdata.39517 2023-01-25T03:05:02.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:02.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj3 testdata.39517 2023-01-25T03:05:04.590 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:04.590 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj4 testdata.39517 2023-01-25T03:05:06.280 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:06.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj5 testdata.39517 2023-01-25T03:05:07.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:07.954 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj6 testdata.39517 2023-01-25T03:05:09.681 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:09.682 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj7 testdata.39517 2023-01-25T03:05:11.294 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:11.294 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj8 testdata.39517 2023-01-25T03:05:12.980 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:12.980 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj9 testdata.39517 2023-01-25T03:05:14.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:14.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj10 testdata.39517 2023-01-25T03:05:16.392 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:16.393 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj11 testdata.39517 2023-01-25T03:05:18.053 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:18.054 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj12 testdata.39517 2023-01-25T03:05:19.763 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:19.763 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj13 testdata.39517 2023-01-25T03:05:21.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:21.319 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj14 testdata.39517 2023-01-25T03:05:22.958 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:22.959 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj15 testdata.39517 2023-01-25T03:05:24.673 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:24.673 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj16 testdata.39517 2023-01-25T03:05:26.296 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:26.296 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj17 testdata.39517 2023-01-25T03:05:27.969 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:27.969 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj18 testdata.39517 2023-01-25T03:05:29.716 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:29.717 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj19 testdata.39517 2023-01-25T03:05:31.264 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:31.265 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj20 testdata.39517 2023-01-25T03:05:32.930 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:32.931 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj21 testdata.39517 2023-01-25T03:05:35.254 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:35.254 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj22 testdata.39517 2023-01-25T03:05:36.893 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:36.893 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj23 testdata.39517 2023-01-25T03:05:38.565 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:38.566 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj24 testdata.39517 2023-01-25T03:05:40.388 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:40.389 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj25 testdata.39517 2023-01-25T03:05:42.120 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:42.120 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj26 testdata.39517 2023-01-25T03:05:43.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:43.976 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj27 testdata.39517 2023-01-25T03:05:45.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:45.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj28 testdata.39517 2023-01-25T03:05:47.621 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:47.622 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj29 testdata.39517 2023-01-25T03:05:49.394 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:49.394 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj30 testdata.39517 2023-01-25T03:05:51.166 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:51.167 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj31 testdata.39517 2023-01-25T03:05:53.023 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:53.024 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj32 testdata.39517 2023-01-25T03:05:54.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:54.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj33 testdata.39517 2023-01-25T03:05:56.368 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:56.369 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj34 testdata.39517 2023-01-25T03:05:58.032 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:58.033 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj35 testdata.39517 2023-01-25T03:05:59.781 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:05:59.781 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj36 testdata.39517 2023-01-25T03:06:01.424 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:01.424 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj37 testdata.39517 2023-01-25T03:06:03.084 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:03.084 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj38 testdata.39517 2023-01-25T03:06:04.957 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:04.957 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj39 testdata.39517 2023-01-25T03:06:06.696 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:06.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj40 testdata.39517 2023-01-25T03:06:08.362 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:08.362 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj41 testdata.39517 2023-01-25T03:06:10.144 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:10.145 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj42 testdata.39517 2023-01-25T03:06:11.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:11.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj43 testdata.39517 2023-01-25T03:06:13.502 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:13.502 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj44 testdata.39517 2023-01-25T03:06:15.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:15.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj45 testdata.39517 2023-01-25T03:06:17.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:17.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj46 testdata.39517 2023-01-25T03:06:18.710 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:18.711 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj47 testdata.39517 2023-01-25T03:06:20.600 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:20.601 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj48 testdata.39517 2023-01-25T03:06:22.432 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:22.432 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj49 testdata.39517 2023-01-25T03:06:24.237 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:24.238 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj50 testdata.39517 2023-01-25T03:06:26.085 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:26.086 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj51 testdata.39517 2023-01-25T03:06:27.758 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:27.759 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj52 testdata.39517 2023-01-25T03:06:29.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:29.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj53 testdata.39517 2023-01-25T03:06:31.112 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:31.112 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj54 testdata.39517 2023-01-25T03:06:32.800 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:32.801 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj55 testdata.39517 2023-01-25T03:06:34.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:34.564 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj56 testdata.39517 2023-01-25T03:06:36.393 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:36.393 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj57 testdata.39517 2023-01-25T03:06:38.120 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:38.120 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj58 testdata.39517 2023-01-25T03:06:39.882 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:39.883 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj59 testdata.39517 2023-01-25T03:06:41.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:41.636 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj60 testdata.39517 2023-01-25T03:06:43.284 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:43.285 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj61 testdata.39517 2023-01-25T03:06:45.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:45.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj62 testdata.39517 2023-01-25T03:06:47.112 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:47.113 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj63 testdata.39517 2023-01-25T03:06:48.767 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:48.768 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj64 testdata.39517 2023-01-25T03:06:50.456 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:50.456 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj65 testdata.39517 2023-01-25T03:06:52.209 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:52.209 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj66 testdata.39517 2023-01-25T03:06:53.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:53.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj67 testdata.39517 2023-01-25T03:06:55.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:55.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj68 testdata.39517 2023-01-25T03:06:57.531 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:57.532 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj69 testdata.39517 2023-01-25T03:06:59.288 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:06:59.289 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj70 testdata.39517 2023-01-25T03:07:01.202 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:01.202 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj71 testdata.39517 2023-01-25T03:07:02.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:02.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj72 testdata.39517 2023-01-25T03:07:04.562 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:04.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj73 testdata.39517 2023-01-25T03:07:06.286 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:06.286 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj74 testdata.39517 2023-01-25T03:07:08.166 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:08.166 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj75 testdata.39517 2023-01-25T03:07:10.026 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:10.027 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj76 testdata.39517 2023-01-25T03:07:11.789 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:11.790 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj77 testdata.39517 2023-01-25T03:07:13.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:13.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj78 testdata.39517 2023-01-25T03:07:15.522 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:15.523 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj79 testdata.39517 2023-01-25T03:07:17.307 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:17.308 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj80 testdata.39517 2023-01-25T03:07:19.084 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:19.084 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj81 testdata.39517 2023-01-25T03:07:20.755 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:20.756 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj82 testdata.39517 2023-01-25T03:07:22.470 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:22.471 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj83 testdata.39517 2023-01-25T03:07:24.131 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:24.132 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj84 testdata.39517 2023-01-25T03:07:25.847 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:25.848 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj85 testdata.39517 2023-01-25T03:07:27.518 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:27.519 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj86 testdata.39517 2023-01-25T03:07:29.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:29.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj87 testdata.39517 2023-01-25T03:07:30.968 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:30.968 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj88 testdata.39517 2023-01-25T03:07:32.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:32.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj89 testdata.39517 2023-01-25T03:07:34.526 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:34.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj90 testdata.39517 2023-01-25T03:07:36.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:36.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj91 testdata.39517 2023-01-25T03:07:38.098 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:38.098 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj92 testdata.39517 2023-01-25T03:07:39.860 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:39.860 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj93 testdata.39517 2023-01-25T03:07:41.701 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:41.702 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj94 testdata.39517 2023-01-25T03:07:43.422 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:43.422 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj95 testdata.39517 2023-01-25T03:07:45.142 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:45.143 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj96 testdata.39517 2023-01-25T03:07:46.805 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:46.805 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj97 testdata.39517 2023-01-25T03:07:48.630 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:48.630 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj98 testdata.39517 2023-01-25T03:07:50.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:50.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj99 testdata.39517 2023-01-25T03:07:52.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-01-25T03:07:52.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj100 testdata.39517 2023-01-25T03:07:53.881 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: rm -f testdata.39517 2023-01-25T03:07:53.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: ceph osd pool set test size 4 2023-01-25T03:07:54.341 INFO:tasks.workunit.client.0.smithi008.stderr:set pool 1 size to 4 2023-01-25T03:07:54.355 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:73: TEST_recovery_scrub_1: set -o pipefail 2023-01-25T03:07:54.355 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: count=0 2023-01-25T03:07:54.356 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T03:07:54.357 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T03:07:54.357 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T03:07:54.358 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T03:07:54.676 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:07:54.686 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2023-01-25T03:07:56.688 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 0 -eq 10 2023-01-25T03:07:56.689 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 0 + 1 2023-01-25T03:07:56.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=1 2023-01-25T03:07:56.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T03:07:56.692 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T03:07:56.693 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T03:07:56.693 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T03:07:57.217 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:07:57.229 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2023-01-25T03:07:59.231 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 1 -eq 10 2023-01-25T03:07:59.232 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 1 + 1 2023-01-25T03:07:59.234 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=2 2023-01-25T03:07:59.234 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T03:07:59.235 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T03:07:59.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T03:07:59.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T03:07:59.520 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:07:59.529 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2023-01-25T03:08:01.530 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 2 -eq 10 2023-01-25T03:08:01.532 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 2 + 1 2023-01-25T03:08:01.533 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=3 2023-01-25T03:08:01.533 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2023-01-25T03:08:01.534 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-01-25T03:08:01.535 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-01-25T03:08:01.535 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2023-01-25T03:08:01.834 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:08:01.844 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:80: TEST_recovery_scrub_1: break 2023-01-25T03:08:01.844 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:90: TEST_recovery_scrub_1: set +o pipefail 2023-01-25T03:08:01.845 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:91: TEST_recovery_scrub_1: ceph pg dump pgs 2023-01-25T03:08:02.138 INFO:tasks.workunit.client.0.smithi008.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T03:08:02.138 INFO:tasks.workunit.client.0.smithi008.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2023-01-25T03:07:56.562887+0000 23'1300 26:1327 [1,0,2,3] 1 [1,0] 1 0'0 2023-01-25T03:04:52.638763+0000 0'0 2023-01-25T03:04:52.638763+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:59:16.632467+0000 0 0 2023-01-25T03:08:02.138 INFO:tasks.workunit.client.0.smithi008.stdout: 2023-01-25T03:08:02.139 INFO:tasks.workunit.client.0.smithi008.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T03:08:02.141 INFO:tasks.workunit.client.0.smithi008.stderr:dumped pgs 2023-01-25T03:08:02.150 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:93: TEST_recovery_scrub_1: sleep 10 2023-01-25T03:08:12.152 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2023-01-25T03:08:12.153 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:08:12.154 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:08:12.155 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:08:12.155 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:08:12.156 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:08:17.483 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:08:17.483 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: declare -a err_strings 2023-01-25T03:08:17.483 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:98: TEST_recovery_scrub_1: err_strings[0]='not scheduling scrubs due to active recovery' 2023-01-25T03:08:17.484 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: expr 4 - 1 2023-01-25T03:08:17.485 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: seq 0 3 2023-01-25T03:08:17.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2023-01-25T03:08:17.493 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:56.825+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.493 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:58.861+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.493 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:01.913+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.494 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:02.925+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.494 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:05.880+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.494 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:06.892+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.494 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:09.880+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.494 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:13.936+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.495 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:14.964+0000 7f09d797d700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2023-01-25T03:08:17.512 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:57.453+0000 7ffa92d7f700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.513 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:59.505+0000 7ffa92d7f700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.513 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:03.408+0000 7ffa92d7f700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.514 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:06.344+0000 7ffa92d7f700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.514 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:09.324+0000 7ffa92d7f700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.514 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:14.316+0000 7ffa92d7f700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.515 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.515 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2023-01-25T03:08:17.517 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:56.841+0000 7f01d9090700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.518 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:58.865+0000 7f01d9090700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.518 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:08.796+0000 7f01d9090700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.518 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:09.772+0000 7f01d9090700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.518 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:11.752+0000 7f01d9090700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.519 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:13.800+0000 7f01d9090700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.520 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.520 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2023-01-25T03:08:17.521 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:07:58.437+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.522 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:03.365+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.522 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:04.348+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.522 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:05.316+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.522 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:09.312+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.522 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:12.356+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.523 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:14.424+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.523 INFO:tasks.workunit.client.0.smithi008.stdout:2023-01-25T03:08:16.436+0000 7f2461b60700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2023-01-25T03:08:17.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2023-01-25T03:08:17.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: found=false 2023-01-25T03:08:17.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:107: TEST_recovery_scrub_1: count=0 2023-01-25T03:08:17.526 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: expr 4 - 1 2023-01-25T03:08:17.526 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: seq 0 3 2023-01-25T03:08:17.526 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.527 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2023-01-25T03:08:17.529 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T03:08:17.531 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 0 + 1 2023-01-25T03:08:17.532 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=1 2023-01-25T03:08:17.532 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.532 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2023-01-25T03:08:17.547 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T03:08:17.547 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 1 + 1 2023-01-25T03:08:17.549 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=2 2023-01-25T03:08:17.552 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.552 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2023-01-25T03:08:17.553 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T03:08:17.554 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 2 + 1 2023-01-25T03:08:17.555 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=3 2023-01-25T03:08:17.555 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:17.555 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2023-01-25T03:08:17.559 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2023-01-25T03:08:17.560 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 3 + 1 2023-01-25T03:08:17.562 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=4 2023-01-25T03:08:17.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: '[' true = false ']' 2023-01-25T03:08:17.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2023-01-25T03:08:17.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:123: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2023-01-25T03:08:17.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-01-25T03:08:17.563 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T03:08:17.564 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-01-25T03:08:17.564 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:08:17.564 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:08:17.565 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:08:17.565 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:08:17.565 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:08:17.575 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:08:17.576 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T03:08:17.576 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T03:08:17.577 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T03:08:17.578 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T03:08:17.579 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T03:08:17.579 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T03:08:17.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:08:17.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T03:08:17.581 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T03:08:17.581 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:08:17.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T03:08:17.584 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T03:08:17.584 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-01-25T03:08:17.617 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T03:08:17.618 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:17.618 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:17.618 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.39517 2023-01-25T03:08:17.619 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T03:08:17.620 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T03:08:17.620 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:125: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2023-01-25T03:08:17.620 INFO:tasks.workunit.client.0.smithi008.stdout:TEST PASSED 2023-01-25T03:08:17.622 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:131: TEST_recovery_scrub_1: echo 'TEST PASSED' 2023-01-25T03:08:17.622 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:132: TEST_recovery_scrub_1: return 0 2023-01-25T03:08:17.623 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2023-01-25T03:08:17.623 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2023-01-25T03:08:17.623 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:230: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2023-01-25T03:08:17.623 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:231: TEST_recovery_scrub_2: local poolname=test 2023-01-25T03:08:17.623 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:233: TEST_recovery_scrub_2: TESTDATA=testdata.39517 2023-01-25T03:08:17.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: OSDS=8 2023-01-25T03:08:17.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: PGS=32 2023-01-25T03:08:17.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:236: TEST_recovery_scrub_2: OBJECTS=40 2023-01-25T03:08:17.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2023-01-25T03:08:17.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-01-25T03:08:17.625 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-01-25T03:08:17.625 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-01-25T03:08:17.625 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T03:08:17.625 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-01-25T03:08:17.625 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T03:08:17.626 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T03:08:17.626 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T03:08:17.626 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T03:08:17.626 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T03:08:17.627 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T03:08:17.627 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T03:08:17.627 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T03:08:17.628 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T03:08:17.629 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T03:08:17.629 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T03:08:17.629 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T03:08:17.631 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:08:17.631 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T03:08:17.632 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T03:08:17.632 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T03:08:17.635 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T03:08:17.637 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T03:08:17.637 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-01-25T03:08:17.638 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T03:08:17.639 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:17.639 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:17.639 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.39517 2023-01-25T03:08:17.640 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T03:08:17.641 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T03:08:17.641 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-01-25T03:08:17.643 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T03:08:17.644 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:17.644 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:17.644 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.39517 2023-01-25T03:08:17.647 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T03:08:17.647 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-01-25T03:08:17.647 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T03:08:17.647 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-01-25T03:08:17.648 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T03:08:17.648 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-01-25T03:08:17.648 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-25T03:08:17.648 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-25T03:08:17.648 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-25T03:08:17.649 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-01-25T03:08:17.649 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T03:08:18.016 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-25T03:08:18.017 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:08:18.017 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:08:18.018 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:08:18.018 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:18.019 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:18.019 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:08:18.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2023-01-25T03:08:18.353 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-25T03:08:18.354 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-25T03:08:18.355 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T03:08:18.355 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T03:08:18.355 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-25T03:08:18.355 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-25T03:08:18.356 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T03:08:18.356 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T03:08:18.357 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T03:08:18.358 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T03:08:18.358 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:18.358 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:18.358 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.39517/ceph-mon.a.asok 2023-01-25T03:08:18.358 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T03:08:18.359 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.39517/ceph-mon.a.asok config get fsid 2023-01-25T03:08:18.454 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-25T03:08:18.454 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T03:08:18.454 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T03:08:18.455 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-25T03:08:18.456 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-25T03:08:18.456 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T03:08:18.456 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T03:08:18.456 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T03:08:18.457 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T03:08:18.457 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:18.457 INFO:tasks.workunit.client.0.smithi008.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:18.457 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.39517/ceph-mon.a.asok 2023-01-25T03:08:18.457 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T03:08:18.458 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.39517/ceph-mon.a.asok config get mon_host 2023-01-25T03:08:18.550 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:241: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2023-01-25T03:08:18.550 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-01-25T03:08:18.550 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-25T03:08:18.551 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-25T03:08:18.551 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-25T03:08:18.551 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-01-25T03:08:18.551 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-25T03:08:19.437 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-25T03:08:19.438 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:08:19.438 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:08:19.440 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:08:19.440 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:19.441 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:19.441 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:08:19.441 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T03:08:19.441 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T03:08:19.482 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: expr 8 - 1 2023-01-25T03:08:19.483 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: seq 0 7 2023-01-25T03:08:19.486 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:19.486 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:08:19.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:08:19.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:08:19.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-25T03:08:19.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:08:19.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-01-25T03:08:19.488 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:08:19.488 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:08:19.488 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:08:19.488 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:08:19.488 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-01-25T03:08:19.489 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-01-25T03:08:19.489 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:08:19.489 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:08:19.489 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:08:19.489 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:08:19.490 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:08:19.490 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:08:19.492 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:08:19.492 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:19.492 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:19.492 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:08:19.493 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:08:19.493 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:08:19.493 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:08:19.493 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:08:19.493 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:08:19.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:08:19.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:08:19.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:08:19.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:08:19.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:08:19.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:08:19.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-01-25T03:08:19.495 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:08:19.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=2800ba82-47b1-41b5-b39e-b38649f4b02a 2023-01-25T03:08:19.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 2800ba82-47b1-41b5-b39e-b38649f4b02a' 2023-01-25T03:08:19.496 INFO:tasks.workunit.client.0.smithi008.stdout:add osd0 2800ba82-47b1-41b5-b39e-b38649f4b02a 2023-01-25T03:08:19.496 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:08:19.512 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAjndBjhSRIHhAAX9s1ZT7qkT+rAGyCQNL2pA== 2023-01-25T03:08:19.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAjndBjhSRIHhAAX9s1ZT7qkT+rAGyCQNL2pA=="}' 2023-01-25T03:08:19.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 2800ba82-47b1-41b5-b39e-b38649f4b02a -i td/osd-recovery-scrub/0/new.json 2023-01-25T03:08:19.798 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:08:19.813 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-01-25T03:08:19.815 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAjndBjhSRIHhAAX9s1ZT7qkT+rAGyCQNL2pA== --osd-uuid 2800ba82-47b1-41b5-b39e-b38649f4b02a 2023-01-25T03:08:19.848 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.840+0000 7f3a88418680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:19.851 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.844+0000 7f3a88418680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:19.853 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.844+0000 7f3a88418680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:19.854 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.844+0000 7f3a88418680 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T03:08:19.854 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.844+0000 7f3a88418680 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T03:08:19.854 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.844+0000 7f3a88418680 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-01-25T03:08:19.923 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:19.916+0000 7f3a88418680 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-01-25T03:08:26.115 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-01-25T03:08:26.115 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:08:26.117 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd0 key to auth repository 2023-01-25T03:08:26.117 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-25T03:08:26.117 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:08:26.494 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.0 2023-01-25T03:08:26.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-25T03:08:26.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:08:26.495 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:08:26.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:08:26.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:08:26.528 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:26.520+0000 7f7c80750680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:26.531 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:26.524+0000 7f7c80750680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:26.534 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:26.524+0000 7f7c80750680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:26.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-25T03:08:26.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:08:26.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T03:08:26.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:08:26.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:08:26.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:26.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:08:26.784 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:08:26.784 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:26.784 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:27.101 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:27.092+0000 7f7c80750680 -1 Falling back to public interface 2023-01-25T03:08:27.185 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:28.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:28.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:28.188 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:08:28.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:08:28.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:28.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:28.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:29.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:29.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:29.497 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:08:29.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:08:29.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:29.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:29.668 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:29.660+0000 7f7c80750680 -1 osd.0 0 log_to_monitors true 2023-01-25T03:08:29.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:30.872 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:30.873 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:30.873 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:08:30.874 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:08:30.874 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:30.874 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:31.213 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:32.215 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:32.215 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:32.215 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:08:32.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:08:32.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:32.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:32.573 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:33.576 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:33.576 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:33.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:08:33.578 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:08:33.578 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:33.579 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:33.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:34.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:34.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:34.891 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:08:34.891 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:08:34.891 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:34.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:35.224 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:36.226 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:36.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:36.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:08:36.227 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:08:36.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:36.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:36.605 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:37.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:37.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:37.608 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:08:37.608 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:08:37.608 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:37.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:37.915 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:38.916 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:38.916 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:38.917 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:08:38.917 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:08:38.918 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:38.918 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:39.288 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:40.290 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:40.290 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:40.291 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:08:40.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:08:40.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:40.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:40.643 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:41.645 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:41.645 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:41.645 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:08:41.646 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:08:41.646 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:41.646 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:41.950 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:42.952 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:42.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:42.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:08:42.953 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:08:42.954 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:42.954 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:43.310 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:44.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:44.312 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:44.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:08:44.313 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:08:44.313 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:44.314 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:44.753 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:45.755 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:45.755 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:45.756 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:08:45.756 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:08:45.757 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:45.757 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:46.254 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:47.256 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:47.256 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:47.256 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:08:47.257 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:08:47.257 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:47.257 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:47.820 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:48.822 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:48.822 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:48.822 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:08:48.823 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:08:48.823 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:48.823 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:49.327 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:50.327 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:50.327 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:50.327 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:08:50.328 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:08:50.328 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:50.328 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:51.053 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:52.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:52.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:52.056 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:08:52.057 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:08:52.057 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:52.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:52.508 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:08:53.510 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:08:53.510 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:08:53.510 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T03:08:53.511 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:08:53.511 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:08:53.512 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T03:08:53.855 INFO:tasks.workunit.client.0.smithi008.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/50727,v1:127.0.0.1:6803/50727] [v2:127.0.0.1:6804/50727,v1:127.0.0.1:6805/50727] exists,up 2800ba82-47b1-41b5-b39e-b38649f4b02a 2023-01-25T03:08:53.858 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:08:53.858 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:08:53.859 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:08:53.860 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:08:53.860 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:08:53.860 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:08:53.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:08:53.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-25T03:08:53.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:08:53.861 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-01-25T03:08:53.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:08:53.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:08:53.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:08:53.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:08:53.862 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-01-25T03:08:53.863 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-01-25T03:08:53.863 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:08:53.863 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:08:53.863 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:08:53.864 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:08:53.864 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:08:53.864 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:08:53.864 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:08:53.865 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:08:53.865 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:08:53.865 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:08:53.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:08:53.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:08:53.867 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:08:53.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:08:53.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:08:53.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:08:53.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:08:53.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:08:53.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:08:53.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:08:53.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:08:53.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-01-25T03:08:53.870 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:08:53.870 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4467e377-8733-4ccd-b96e-a4fac93fa602 2023-01-25T03:08:53.870 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 4467e377-8733-4ccd-b96e-a4fac93fa602' 2023-01-25T03:08:53.871 INFO:tasks.workunit.client.0.smithi008.stdout:add osd1 4467e377-8733-4ccd-b96e-a4fac93fa602 2023-01-25T03:08:53.871 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:08:53.891 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBFndBj/BHUNBAAlxczDfs3VYJv3Gd7aTJLzA== 2023-01-25T03:08:53.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBFndBj/BHUNBAAlxczDfs3VYJv3Gd7aTJLzA=="}' 2023-01-25T03:08:53.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4467e377-8733-4ccd-b96e-a4fac93fa602 -i td/osd-recovery-scrub/1/new.json 2023-01-25T03:08:54.299 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:08:54.316 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-01-25T03:08:54.318 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBFndBj/BHUNBAAlxczDfs3VYJv3Gd7aTJLzA== --osd-uuid 4467e377-8733-4ccd-b96e-a4fac93fa602 2023-01-25T03:08:54.356 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.343+0000 7f1b96d76680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:54.359 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.347+0000 7f1b96d76680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:54.361 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.351+0000 7f1b96d76680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:08:54.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.351+0000 7f1b96d76680 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T03:08:54.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.351+0000 7f1b96d76680 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T03:08:54.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.351+0000 7f1b96d76680 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-01-25T03:08:54.472 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:08:54.463+0000 7f1b96d76680 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-01-25T03:09:01.090 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-01-25T03:09:01.091 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:09:01.093 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd1 key to auth repository 2023-01-25T03:09:01.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-25T03:09:01.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:09:01.474 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.1 2023-01-25T03:09:01.475 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-25T03:09:01.475 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:09:01.475 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:09:01.476 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:09:01.476 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:09:01.511 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:01.499+0000 7efd8f9c1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:01.536 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:01.527+0000 7efd8f9c1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:01.538 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:01.527+0000 7efd8f9c1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:01.816 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-25T03:09:01.817 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:09:01.817 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T03:09:01.817 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:09:01.817 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:09:01.818 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:01.818 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:09:01.818 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:09:01.819 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:01.819 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:02.089 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:02.079+0000 7efd8f9c1680 -1 Falling back to public interface 2023-01-25T03:09:02.114 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:03.116 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:03.116 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:03.117 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:09:03.117 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:09:03.118 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:03.118 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:03.431 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:04.433 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:04.433 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:04.434 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:09:04.434 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:09:04.435 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:04.435 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:04.679 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:04.671+0000 7efd8f9c1680 -1 osd.1 0 log_to_monitors true 2023-01-25T03:09:04.814 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:05.816 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:05.816 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:05.817 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:09:05.817 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:09:05.818 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:05.818 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:06.332 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:07.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:07.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:07.335 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:09:07.335 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:09:07.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:07.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:07.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:08.696 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:08.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:08.697 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:09:08.698 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:09:08.698 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:08.698 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:09.027 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:10.028 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:10.028 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:10.029 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:09:10.030 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:09:10.030 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:10.030 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:10.363 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:11.364 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:11.365 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:11.365 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:09:11.365 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:09:11.366 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:11.366 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:11.753 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:12.755 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:12.756 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:12.756 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:09:12.756 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:09:12.757 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:12.757 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:13.068 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:14.070 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:14.070 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:14.070 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:09:14.071 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:09:14.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:14.071 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:14.379 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:15.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:15.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:15.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:09:15.382 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:09:15.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:15.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:15.699 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:16.701 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:16.701 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:16.701 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:09:16.702 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:09:16.702 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:16.702 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:17.033 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:18.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:18.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:18.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:09:18.036 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:09:18.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:18.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:18.522 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:19.523 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:19.524 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:19.524 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:09:19.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:09:19.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:19.525 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:19.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:20.904 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:20.904 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:20.904 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:09:20.905 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:09:20.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:20.906 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:21.397 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:22.399 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:22.399 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:22.400 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:09:22.400 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:09:22.400 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:22.401 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:22.838 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:23.839 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:23.840 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:23.840 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:09:23.841 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:09:23.841 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:23.841 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:24.272 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:25.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:25.274 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:25.274 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:09:25.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:09:25.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:25.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:25.938 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:26.940 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:26.940 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:26.941 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:09:26.941 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:09:26.941 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:26.942 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:27.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:28.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:28.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:28.336 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:09:28.337 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T03:09:28.337 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:28.338 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:28.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:29.696 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:29.696 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:29.696 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 20 2023-01-25T03:09:29.697 INFO:tasks.workunit.client.0.smithi008.stdout:20 2023-01-25T03:09:29.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:29.697 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T03:09:30.005 INFO:tasks.workunit.client.0.smithi008.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/52515,v1:127.0.0.1:6811/52515] [v2:127.0.0.1:6812/52515,v1:127.0.0.1:6813/52515] exists,up 4467e377-8733-4ccd-b96e-a4fac93fa602 2023-01-25T03:09:30.007 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:09:30.007 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:09:30.007 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:09:30.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:09:30.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:09:30.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:09:30.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:09:30.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-25T03:09:30.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:09:30.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-01-25T03:09:30.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:09:30.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:09:30.009 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:09:30.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:09:30.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-01-25T03:09:30.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-01-25T03:09:30.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:09:30.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:09:30.011 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:09:30.011 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:09:30.011 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:09:30.011 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:09:30.013 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:09:30.013 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:09:30.013 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:09:30.014 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:09:30.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:09:30.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:09:30.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:09:30.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:09:30.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:09:30.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:09:30.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:09:30.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:09:30.015 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:09:30.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:09:30.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:09:30.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-01-25T03:09:30.029 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:09:30.031 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1e6aa7c9-fedf-4abc-a80b-650b84bcb795 2023-01-25T03:09:30.031 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 1e6aa7c9-fedf-4abc-a80b-650b84bcb795' 2023-01-25T03:09:30.032 INFO:tasks.workunit.client.0.smithi008.stdout:add osd2 1e6aa7c9-fedf-4abc-a80b-650b84bcb795 2023-01-25T03:09:30.032 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:09:30.058 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBqndBjUkgsAxAA9pHZen2v7IxdNwNOFzHYKw== 2023-01-25T03:09:30.059 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBqndBjUkgsAxAA9pHZen2v7IxdNwNOFzHYKw=="}' 2023-01-25T03:09:30.059 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1e6aa7c9-fedf-4abc-a80b-650b84bcb795 -i td/osd-recovery-scrub/2/new.json 2023-01-25T03:09:30.437 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:09:30.454 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-01-25T03:09:30.456 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBqndBjUkgsAxAA9pHZen2v7IxdNwNOFzHYKw== --osd-uuid 1e6aa7c9-fedf-4abc-a80b-650b84bcb795 2023-01-25T03:09:30.495 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.486+0000 7f022249f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:30.498 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.490+0000 7f022249f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:30.505 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.494+0000 7f022249f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:30.505 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.494+0000 7f022249f680 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T03:09:30.505 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.494+0000 7f022249f680 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T03:09:30.505 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.494+0000 7f022249f680 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-01-25T03:09:30.552 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:30.546+0000 7f022249f680 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-01-25T03:09:37.171 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-01-25T03:09:37.172 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:09:37.173 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd2 key to auth repository 2023-01-25T03:09:37.174 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-25T03:09:37.174 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:09:37.560 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.2 2023-01-25T03:09:37.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-25T03:09:37.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:09:37.562 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:09:37.562 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:09:37.562 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:09:37.594 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:37.586+0000 7f4700532680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:37.596 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:37.590+0000 7f4700532680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:37.598 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:37.590+0000 7f4700532680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:09:37.852 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-25T03:09:37.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:09:37.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T03:09:37.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:09:37.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:09:37.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:37.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:09:37.854 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:09:37.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:37.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:38.150 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:38.142+0000 7f4700532680 -1 Falling back to public interface 2023-01-25T03:09:38.184 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:39.185 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:39.186 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:39.186 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:09:39.186 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:09:39.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:39.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:39.501 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:40.503 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:40.503 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:40.504 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:09:40.504 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:09:40.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:40.505 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:40.771 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:09:40.762+0000 7f4700532680 -1 osd.2 0 log_to_monitors true 2023-01-25T03:09:40.817 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:41.819 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:41.819 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:41.819 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:09:41.820 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:09:41.820 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:41.821 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:42.183 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:43.185 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:43.185 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:43.186 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:09:43.186 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:09:43.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:43.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:43.518 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:44.520 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:44.520 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:44.520 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:09:44.521 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:09:44.521 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:44.521 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:44.892 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:45.894 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:45.894 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:45.894 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:09:45.895 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:09:45.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:45.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:46.213 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:47.215 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:47.215 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:47.215 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:09:47.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:09:47.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:47.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:47.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:48.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:48.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:48.610 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:09:48.610 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:09:48.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:48.611 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:48.923 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:49.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:49.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:49.925 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:09:49.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:09:49.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:49.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:50.352 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:51.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:51.354 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:51.355 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:09:51.355 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:09:51.356 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:51.356 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:51.832 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:52.835 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:52.835 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:52.835 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:09:52.836 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:09:52.836 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:52.837 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:53.183 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:54.185 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:54.186 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:54.187 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:09:54.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:09:54.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:54.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:54.667 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:55.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:55.670 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:55.670 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:09:55.671 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:09:55.671 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:55.671 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:55.983 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:56.985 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:56.986 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:56.986 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:09:56.987 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:09:56.987 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:56.987 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:57.747 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:09:58.749 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:09:58.749 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:09:58.750 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:09:58.750 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:09:58.750 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:09:58.751 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:09:59.570 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:00.572 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:00.573 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:00.573 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:10:00.573 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:10:00.574 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:00.574 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:10:01.032 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:02.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:02.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:02.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:10:02.036 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:10:02.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:02.036 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:10:02.847 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:03.849 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:03.849 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:03.849 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:10:03.850 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:10:03.850 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:03.851 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T03:10:04.183 INFO:tasks.workunit.client.0.smithi008.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54328,v1:127.0.0.1:6819/54328] [v2:127.0.0.1:6820/54328,v1:127.0.0.1:6821/54328] exists,up 1e6aa7c9-fedf-4abc-a80b-650b84bcb795 2023-01-25T03:10:04.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:10:04.187 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:10:04.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:10:04.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:10:04.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:10:04.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:10:04.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:10:04.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-01-25T03:10:04.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:10:04.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-01-25T03:10:04.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:10:04.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:10:04.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:10:04.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:10:04.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-01-25T03:10:04.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-01-25T03:10:04.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:10:04.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:10:04.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:10:04.191 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:10:04.191 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:10:04.192 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:10:04.192 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:10:04.192 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:10:04.192 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:10:04.192 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:10:04.193 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:10:04.193 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:10:04.193 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:10:04.193 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:10:04.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:10:04.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:10:04.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:10:04.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:10:04.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:10:04.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:10:04.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:10:04.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-01-25T03:10:04.196 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:10:04.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=3a2cf22b-a961-4b01-bd1f-bfe517e688ed 2023-01-25T03:10:04.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 3a2cf22b-a961-4b01-bd1f-bfe517e688ed' 2023-01-25T03:10:04.197 INFO:tasks.workunit.client.0.smithi008.stdout:add osd3 3a2cf22b-a961-4b01-bd1f-bfe517e688ed 2023-01-25T03:10:04.197 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:10:04.216 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCMndBjepeLDBAABatIQDm0Kj0KK6ZRz2bBiw== 2023-01-25T03:10:04.217 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCMndBjepeLDBAABatIQDm0Kj0KK6ZRz2bBiw=="}' 2023-01-25T03:10:04.217 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 3a2cf22b-a961-4b01-bd1f-bfe517e688ed -i td/osd-recovery-scrub/3/new.json 2023-01-25T03:10:04.619 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:10:04.634 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-01-25T03:10:04.636 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQCMndBjepeLDBAABatIQDm0Kj0KK6ZRz2bBiw== --osd-uuid 3a2cf22b-a961-4b01-bd1f-bfe517e688ed 2023-01-25T03:10:04.675 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.666+0000 7f628870d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:04.679 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.674+0000 7f628870d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:04.681 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.674+0000 7f628870d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:04.682 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.674+0000 7f628870d680 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T03:10:04.683 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.674+0000 7f628870d680 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T03:10:04.683 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.674+0000 7f628870d680 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-01-25T03:10:04.747 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:04.738+0000 7f628870d680 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-01-25T03:10:11.256 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-01-25T03:10:11.257 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:10:11.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2023-01-25T03:10:11.261 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd3 key to auth repository 2023-01-25T03:10:11.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:10:11.705 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.3 2023-01-25T03:10:11.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2023-01-25T03:10:11.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:10:11.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:10:11.706 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:10:11.707 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:10:11.740 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:11.734+0000 7fa2a90d7680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:11.742 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:11.734+0000 7fa2a90d7680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:11.744 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:11.738+0000 7fa2a90d7680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:12.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2023-01-25T03:10:12.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:10:12.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=3 2023-01-25T03:10:12.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:10:12.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:10:12.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:12.019 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:10:12.019 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:10:12.020 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:12.020 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:12.302 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:12.294+0000 7fa2a90d7680 -1 Falling back to public interface 2023-01-25T03:10:12.361 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:13.362 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:13.363 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:13.363 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:10:13.363 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:10:13.364 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:13.364 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:13.667 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:14.404 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:14.397+0000 7fa2a90d7680 -1 osd.3 0 log_to_monitors true 2023-01-25T03:10:14.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:14.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:14.669 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:10:14.670 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:10:14.670 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:14.670 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:15.177 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:16.179 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:16.179 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:16.180 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:10:16.180 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:10:16.181 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:16.181 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:16.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:17.515 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:17.515 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:17.516 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:10:17.516 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:10:17.517 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:17.517 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:17.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:18.845 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:18.845 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:18.845 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:10:18.846 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:10:18.846 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:18.846 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:19.259 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:20.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:20.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:20.262 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:10:20.262 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:10:20.262 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:20.263 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:20.575 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:21.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:21.650 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:21.650 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:10:21.651 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:10:21.651 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:21.651 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:21.885 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:22.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:22.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:22.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:10:22.889 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:10:22.889 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:22.889 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:23.281 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:24.283 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:24.283 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:24.283 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:10:24.284 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:10:24.284 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:24.285 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:24.603 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:25.605 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:25.606 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:25.606 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:10:25.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:10:25.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:25.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:26.244 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:27.246 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:27.247 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:27.247 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:10:27.248 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:10:27.248 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:27.248 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:27.556 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:28.559 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:28.559 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:28.560 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:10:28.560 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:10:28.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:28.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:29.192 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:30.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:30.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:30.194 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:10:30.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:10:30.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:30.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:30.876 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:31.878 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:31.879 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:31.879 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:10:31.880 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:10:31.880 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:31.880 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:32.428 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:33.429 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:33.430 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:33.430 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:10:33.431 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:10:33.431 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:33.431 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:33.794 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:34.796 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:34.796 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:34.797 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:10:34.797 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:10:34.797 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:34.798 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:35.367 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:36.369 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:36.370 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:36.370 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:10:36.370 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:10:36.371 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:36.371 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:37.261 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:38.263 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:38.263 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:38.264 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:10:38.264 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:10:38.264 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:38.265 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:38.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:39.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:39.580 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:39.581 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:10:39.581 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T03:10:39.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:39.582 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T03:10:39.891 INFO:tasks.workunit.client.0.smithi008.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/56093,v1:127.0.0.1:6827/56093] [v2:127.0.0.1:6828/56093,v1:127.0.0.1:6829/56093] exists,up 3a2cf22b-a961-4b01-bd1f-bfe517e688ed 2023-01-25T03:10:39.894 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:10:39.894 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:10:39.894 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:10:39.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:10:39.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:10:39.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:10:39.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:10:39.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-01-25T03:10:39.895 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:10:39.896 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2023-01-25T03:10:39.896 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:10:39.896 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:10:39.896 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:10:39.897 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:10:39.897 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2023-01-25T03:10:39.897 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2023-01-25T03:10:39.898 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:10:39.898 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:10:39.898 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:10:39.898 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:10:39.898 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:10:39.899 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:10:39.899 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:10:39.899 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:10:39.899 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:10:39.899 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:10:39.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:10:39.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:10:39.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:10:39.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:10:39.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:10:39.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:10:39.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:10:39.903 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:10:39.903 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:10:39.903 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:10:39.903 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:10:39.903 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/4 2023-01-25T03:10:39.904 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:10:39.904 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=3020b7c1-8aac-477e-aa61-cb7e417a1c36 2023-01-25T03:10:39.904 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 3020b7c1-8aac-477e-aa61-cb7e417a1c36' 2023-01-25T03:10:39.904 INFO:tasks.workunit.client.0.smithi008.stdout:add osd4 3020b7c1-8aac-477e-aa61-cb7e417a1c36 2023-01-25T03:10:39.905 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:10:39.924 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCvndBjSM+9NhAAu8D8kHMQzEARipJFX+waCg== 2023-01-25T03:10:39.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCvndBjSM+9NhAAu8D8kHMQzEARipJFX+waCg=="}' 2023-01-25T03:10:39.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 3020b7c1-8aac-477e-aa61-cb7e417a1c36 -i td/osd-recovery-scrub/4/new.json 2023-01-25T03:10:40.304 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:10:40.322 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/4/new.json 2023-01-25T03:10:40.324 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQCvndBjSM+9NhAAu8D8kHMQzEARipJFX+waCg== --osd-uuid 3020b7c1-8aac-477e-aa61-cb7e417a1c36 2023-01-25T03:10:40.354 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.345+0000 7faba45c2680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:40.358 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.349+0000 7faba45c2680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:40.361 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.353+0000 7faba45c2680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:40.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.353+0000 7faba45c2680 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-01-25T03:10:40.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.353+0000 7faba45c2680 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-01-25T03:10:40.362 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.353+0000 7faba45c2680 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-01-25T03:10:40.477 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:40.469+0000 7faba45c2680 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2023-01-25T03:10:47.116 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2023-01-25T03:10:47.117 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:10:47.118 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd4 key to auth repository 2023-01-25T03:10:47.118 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2023-01-25T03:10:47.119 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:10:47.511 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.4 2023-01-25T03:10:47.512 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2023-01-25T03:10:47.512 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:10:47.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:10:47.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:10:47.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:10:47.545 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:47.537+0000 7f88c18c8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:47.549 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:47.541+0000 7f88c18c8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:47.551 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:47.541+0000 7f88c18c8680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:10:47.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2023-01-25T03:10:47.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:10:47.853 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=4 2023-01-25T03:10:47.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:10:47.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:10:47.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:47.854 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:10:47.855 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:10:47.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:47.855 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:48.118 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:48.109+0000 7f88c18c8680 -1 Falling back to public interface 2023-01-25T03:10:48.205 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:49.207 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:49.208 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:49.208 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:10:49.208 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:10:49.209 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:49.209 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:49.513 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:50.515 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:50.516 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:50.516 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:10:50.516 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:10:50.517 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:50.517 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:50.645 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:10:50.637+0000 7f88c18c8680 -1 osd.4 0 log_to_monitors true 2023-01-25T03:10:50.874 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:51.875 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:51.876 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:51.876 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:10:51.877 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:10:51.877 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:51.877 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:52.332 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:53.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:53.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:53.335 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:10:53.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:10:53.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:53.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:53.837 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:54.839 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:54.840 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:54.840 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:10:54.841 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:10:54.841 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:54.842 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:55.149 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:56.151 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:56.151 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:56.152 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:10:56.152 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:10:56.152 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:56.153 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:56.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:57.499 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:57.500 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:57.500 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:10:57.501 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:10:57.501 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:57.502 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:57.873 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:10:58.875 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:10:58.876 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:10:58.876 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:10:58.877 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:10:58.877 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:10:58.877 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:10:59.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:00.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:00.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:00.196 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:11:00.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:11:00.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:00.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:00.627 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:01.629 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:01.629 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:01.629 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:11:01.630 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:11:01.630 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:01.630 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:01.941 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:02.943 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:02.943 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:02.944 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:11:02.944 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:11:02.945 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:02.945 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:03.378 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:04.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:04.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:04.381 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:11:04.382 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:11:04.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:04.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:05.008 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:06.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:06.010 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:06.011 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:11:06.011 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:11:06.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:06.012 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:06.731 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:07.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:07.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:07.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:11:07.735 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:11:07.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:07.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:08.557 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:09.559 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:09.560 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:09.560 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:11:09.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:11:09.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:09.561 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:09.929 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:10.931 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:10.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:10.932 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:11:10.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:11:10.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:10.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:11.471 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:12.473 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:12.473 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:12.473 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:11:12.474 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:11:12.474 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:12.474 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:12.930 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:13.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:13.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:13.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:11:13.933 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:11:13.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:13.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:14.443 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:15.445 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:15.445 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:15.445 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:11:15.446 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T03:11:15.446 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:15.446 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T03:11:15.771 INFO:tasks.workunit.client.0.smithi008.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/57881,v1:127.0.0.1:6835/57881] [v2:127.0.0.1:6836/57881,v1:127.0.0.1:6837/57881] exists,up 3020b7c1-8aac-477e-aa61-cb7e417a1c36 2023-01-25T03:11:15.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:11:15.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:11:15.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:11:15.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:11:15.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:11:15.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:11:15.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:11:15.776 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2023-01-25T03:11:15.776 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:11:15.776 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2023-01-25T03:11:15.776 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:11:15.777 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:11:15.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:11:15.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:11:15.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2023-01-25T03:11:15.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2023-01-25T03:11:15.778 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:11:15.779 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:11:15.779 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:11:15.779 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:11:15.779 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:11:15.779 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:11:15.780 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:11:15.780 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:11:15.780 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:11:15.780 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:11:15.781 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:11:15.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:11:15.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:11:15.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:11:15.782 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:11:15.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:11:15.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:11:15.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:11:15.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:11:15.783 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:11:15.784 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:11:15.784 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/5 2023-01-25T03:11:15.784 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:11:15.785 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=aee1acb9-f5f7-4277-82c5-76ba6112986e 2023-01-25T03:11:15.785 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd5 aee1acb9-f5f7-4277-82c5-76ba6112986e' 2023-01-25T03:11:15.785 INFO:tasks.workunit.client.0.smithi008.stdout:add osd5 aee1acb9-f5f7-4277-82c5-76ba6112986e 2023-01-25T03:11:15.786 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:11:15.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDTndBj0iC2LxAAs+53dXfDoZtvVJA8EPBpeg== 2023-01-25T03:11:15.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDTndBj0iC2LxAAs+53dXfDoZtvVJA8EPBpeg=="}' 2023-01-25T03:11:15.807 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new aee1acb9-f5f7-4277-82c5-76ba6112986e -i td/osd-recovery-scrub/5/new.json 2023-01-25T03:11:16.280 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:11:16.300 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/5/new.json 2023-01-25T03:11:16.302 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 5 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQDTndBj0iC2LxAAs+53dXfDoZtvVJA8EPBpeg== --osd-uuid aee1acb9-f5f7-4277-82c5-76ba6112986e 2023-01-25T03:11:16.336 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.324+0000 7f277e519680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:16.339 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.328+0000 7f277e519680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:16.341 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.332+0000 7f277e519680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:16.342 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.332+0000 7f277e519680 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-01-25T03:11:16.343 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.332+0000 7f277e519680 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-01-25T03:11:16.343 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.332+0000 7f277e519680 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-01-25T03:11:16.405 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:16.392+0000 7f277e519680 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2023-01-25T03:11:23.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2023-01-25T03:11:23.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:11:23.230 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd5 key to auth repository 2023-01-25T03:11:23.231 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd5 key to auth repository 2023-01-25T03:11:23.231 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:11:23.622 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.5 2023-01-25T03:11:23.623 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.5 2023-01-25T03:11:23.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 5 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:11:23.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:11:23.624 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:11:23.625 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:11:23.650 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:23.640+0000 7f237c51b680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:23.679 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:23.668+0000 7f237c51b680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:23.680 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:23.672+0000 7f237c51b680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:23.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 5 2023-01-25T03:11:23.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:11:23.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=5 2023-01-25T03:11:23.905 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:11:23.906 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:11:23.906 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:23.906 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:11:23.906 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:11:23.907 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:23.907 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:24.219 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:24.208+0000 7f237c51b680 -1 Falling back to public interface 2023-01-25T03:11:24.275 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:25.277 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:25.278 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:25.278 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:11:25.278 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:11:25.279 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:25.279 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:25.618 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:26.620 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:26.621 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:26.621 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:11:26.621 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:11:26.622 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:26.622 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:26.787 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:26.776+0000 7f237c51b680 -1 osd.5 0 log_to_monitors true 2023-01-25T03:11:27.004 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:28.006 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:28.006 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:28.006 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:11:28.007 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:11:28.007 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:28.007 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:28.334 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:29.336 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:29.337 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:29.337 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:11:29.338 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:11:29.338 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:29.338 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:29.659 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:30.661 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:30.661 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:30.662 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:11:30.662 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:11:30.662 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:30.663 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:30.975 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:31.977 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:31.977 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:31.978 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:11:31.978 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:11:31.979 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:31.979 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:32.455 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:33.457 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:33.458 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:33.458 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:11:33.458 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:11:33.459 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:33.459 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:33.945 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:34.947 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:34.948 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:34.948 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:11:34.949 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:11:34.949 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:34.949 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:35.335 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:36.337 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:36.337 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:36.338 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:11:36.338 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:11:36.338 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:36.339 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:36.810 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:37.812 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:37.812 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:37.813 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:11:37.813 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:11:37.814 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:37.814 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:38.222 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:39.224 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:39.224 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:39.225 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:11:39.226 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:11:39.226 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:39.227 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:39.829 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:40.833 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:40.833 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:40.834 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:11:40.835 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:11:40.835 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:40.835 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:41.571 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:42.573 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:42.573 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:42.574 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:11:42.574 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:11:42.575 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:42.575 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:42.898 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:43.899 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:43.900 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:43.900 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:11:43.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:11:43.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:43.902 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:44.364 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:45.366 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:45.367 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:45.367 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:11:45.368 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:11:45.368 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:45.369 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:46.042 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:47.044 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:47.044 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:47.044 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:11:47.045 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:11:47.045 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:47.046 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:47.709 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:48.710 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:48.711 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:48.711 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:11:48.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:11:48.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:48.712 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:49.232 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:50.235 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:50.235 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:50.235 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:11:50.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:11:50.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:50.237 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:50.585 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:11:51.587 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:11:51.588 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:51.588 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T03:11:51.588 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:11:51.589 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:51.589 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.5 up' 2023-01-25T03:11:51.923 INFO:tasks.workunit.client.0.smithi008.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/59669,v1:127.0.0.1:6843/59669] [v2:127.0.0.1:6844/59669,v1:127.0.0.1:6845/59669] exists,up aee1acb9-f5f7-4277-82c5-76ba6112986e 2023-01-25T03:11:51.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:11:51.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:11:51.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:11:51.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:11:51.925 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:11:51.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:11:51.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:11:51.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2023-01-25T03:11:51.926 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:11:51.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2023-01-25T03:11:51.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:11:51.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:11:51.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:11:51.927 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:11:51.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2023-01-25T03:11:51.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2023-01-25T03:11:51.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:11:51.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:11:51.928 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:11:51.929 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:11:51.929 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:11:51.929 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:11:51.931 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:11:51.931 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:11:51.931 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:11:51.932 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:11:51.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:11:51.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:11:51.932 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:11:51.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:11:51.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:11:51.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:11:51.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:11:51.933 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:11:51.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:11:51.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:11:51.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:11:51.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/6 2023-01-25T03:11:51.934 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:11:51.934 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=dbbe22d6-78fd-4330-8409-f1ae8a0256bf 2023-01-25T03:11:51.935 INFO:tasks.workunit.client.0.smithi008.stdout:add osd6 dbbe22d6-78fd-4330-8409-f1ae8a0256bf 2023-01-25T03:11:51.935 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd6 dbbe22d6-78fd-4330-8409-f1ae8a0256bf' 2023-01-25T03:11:51.936 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:11:51.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD3ndBjfLRfOBAAc9FFPUikvLGlCwN7SrlPlQ== 2023-01-25T03:11:51.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD3ndBjfLRfOBAAc9FFPUikvLGlCwN7SrlPlQ=="}' 2023-01-25T03:11:51.953 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new dbbe22d6-78fd-4330-8409-f1ae8a0256bf -i td/osd-recovery-scrub/6/new.json 2023-01-25T03:11:52.410 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:11:52.431 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/6/new.json 2023-01-25T03:11:52.432 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 6 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQD3ndBjfLRfOBAAc9FFPUikvLGlCwN7SrlPlQ== --osd-uuid dbbe22d6-78fd-4330-8409-f1ae8a0256bf 2023-01-25T03:11:52.470 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.459+0000 7f2ac0f1d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:52.474 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.463+0000 7f2ac0f1d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:52.476 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.463+0000 7f2ac0f1d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:52.477 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.467+0000 7f2ac0f1d680 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-01-25T03:11:52.478 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.467+0000 7f2ac0f1d680 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-01-25T03:11:52.478 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.467+0000 7f2ac0f1d680 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-01-25T03:11:52.574 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:52.563+0000 7f2ac0f1d680 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2023-01-25T03:11:59.194 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2023-01-25T03:11:59.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:11:59.196 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd6 key to auth repository 2023-01-25T03:11:59.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd6 key to auth repository 2023-01-25T03:11:59.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:11:59.576 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.6 2023-01-25T03:11:59.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.6 2023-01-25T03:11:59.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:11:59.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:11:59.577 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 6 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:11:59.578 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:11:59.611 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:59.599+0000 7f71c99cb680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:59.616 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:59.603+0000 7f71c99cb680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:59.618 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:11:59.607+0000 7f71c99cb680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:11:59.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 6 2023-01-25T03:11:59.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:11:59.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=6 2023-01-25T03:11:59.888 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:11:59.889 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:11:59.889 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:11:59.889 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:11:59.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:11:59.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:11:59.890 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:00.163 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:00.151+0000 7f71c99cb680 -1 Falling back to public interface 2023-01-25T03:12:00.234 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:01.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:01.236 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:01.237 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:12:01.237 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:12:01.237 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:01.238 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:01.556 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:02.468 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:02.455+0000 7f71c99cb680 -1 osd.6 0 log_to_monitors true 2023-01-25T03:12:02.558 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:02.559 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:02.559 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:12:02.560 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:12:02.560 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:02.560 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:02.967 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:03.969 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:03.970 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:03.970 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:12:03.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:12:03.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:03.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:04.290 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:05.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:05.292 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:05.292 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:12:05.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:12:05.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:05.293 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:05.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:06.692 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:06.693 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:06.693 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:12:06.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:12:06.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:06.694 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:07.014 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:08.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:08.016 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:08.017 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:12:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:12:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:08.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:08.327 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:09.329 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:09.330 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:09.330 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:12:09.331 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:12:09.331 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:09.332 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:09.643 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:10.646 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:10.646 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:10.646 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:12:10.647 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:12:10.647 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:10.647 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:10.969 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:11.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:11.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:11.971 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:12:11.972 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:12:11.972 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:11.972 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:12.280 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:13.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:13.282 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:13.282 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:12:13.283 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:12:13.284 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:13.284 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:13.678 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:14.680 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:14.680 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:14.681 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:12:14.681 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:12:14.681 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:14.682 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:14.996 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:15.998 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:15.998 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:15.999 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:12:15.999 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:12:16.000 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:16.000 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:16.614 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:17.616 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:17.616 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:17.617 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:12:17.617 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:12:17.617 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:17.618 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:18.494 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:19.496 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:19.497 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:19.497 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:12:19.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:12:19.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:19.498 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:20.165 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:21.167 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:21.168 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:21.168 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:12:21.169 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:12:21.169 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:21.169 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:21.682 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:22.684 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:22.684 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:22.684 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:12:22.685 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:12:22.685 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:22.686 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:23.240 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:24.242 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:24.242 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:24.242 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:12:24.243 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:12:24.243 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:24.243 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:24.873 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:25.875 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:25.875 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:25.875 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:12:25.876 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:12:25.876 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:25.876 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.6 up' 2023-01-25T03:12:26.186 INFO:tasks.workunit.client.0.smithi008.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/61458,v1:127.0.0.1:6851/61458] [v2:127.0.0.1:6852/61458,v1:127.0.0.1:6853/61458] exists,up dbbe22d6-78fd-4330-8409-f1ae8a0256bf 2023-01-25T03:12:26.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:12:26.188 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:12:26.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:12:26.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T03:12:26.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:12:26.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-01-25T03:12:26.189 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T03:12:26.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2023-01-25T03:12:26.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T03:12:26.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2023-01-25T03:12:26.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 ' 2023-01-25T03:12:26.190 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T03:12:26.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T03:12:26.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T03:12:26.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2023-01-25T03:12:26.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2023-01-25T03:12:26.191 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T03:12:26.192 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T03:12:26.192 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-01-25T03:12:26.192 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T03:12:26.192 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T03:12:26.193 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T03:12:26.194 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T03:12:26.195 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T03:12:26.195 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.39517 2023-01-25T03:12:26.195 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:12:26.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' 2023-01-25T03:12:26.195 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T03:12:26.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T03:12:26.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T03:12:26.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-01-25T03:12:26.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-01-25T03:12:26.196 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T03:12:26.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T03:12:26.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T03:12:26.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T03:12:26.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2023-01-25T03:12:26.197 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-scrub/7 2023-01-25T03:12:26.198 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T03:12:26.198 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=b6f3d4ad-ad20-4a13-a640-fc38c165e64e 2023-01-25T03:12:26.198 INFO:tasks.workunit.client.0.smithi008.stdout:add osd7 b6f3d4ad-ad20-4a13-a640-fc38c165e64e 2023-01-25T03:12:26.199 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd7 b6f3d4ad-ad20-4a13-a640-fc38c165e64e' 2023-01-25T03:12:26.199 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T03:12:26.219 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAantBjFHigDBAAzD8OAL/Ks4LS8G6f/B8Odg== 2023-01-25T03:12:26.219 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAantBjFHigDBAAzD8OAL/Ks4LS8G6f/B8Odg=="}' 2023-01-25T03:12:26.219 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new b6f3d4ad-ad20-4a13-a640-fc38c165e64e -i td/osd-recovery-scrub/7/new.json 2023-01-25T03:12:26.734 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:12:26.755 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-scrub/7/new.json 2023-01-25T03:12:26.757 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 7 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAantBjFHigDBAAzD8OAL/Ks4LS8G6f/B8Odg== --osd-uuid b6f3d4ad-ad20-4a13-a640-fc38c165e64e 2023-01-25T03:12:26.792 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.783+0000 7fcbda556680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:12:26.871 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.863+0000 7fcbda556680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:12:26.873 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.867+0000 7fcbda556680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:12:26.874 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.867+0000 7fcbda556680 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-01-25T03:12:26.874 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.867+0000 7fcbda556680 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-01-25T03:12:26.875 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.867+0000 7fcbda556680 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-01-25T03:12:26.940 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:26.931+0000 7fcbda556680 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2023-01-25T03:12:33.767 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2023-01-25T03:12:33.767 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T03:12:33.769 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd7 key to auth repository 2023-01-25T03:12:33.769 INFO:tasks.workunit.client.0.smithi008.stdout:adding osd7 key to auth repository 2023-01-25T03:12:33.770 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T03:12:34.152 INFO:tasks.workunit.client.0.smithi008.stdout:start osd.7 2023-01-25T03:12:34.153 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.7 2023-01-25T03:12:34.153 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 7 --fsid=91b026b0-97a9-4caa-b024-79d2482a14fc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.39517/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2023-01-25T03:12:34.153 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T03:12:34.154 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T03:12:34.154 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T03:12:34.186 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:34.179+0000 7f50cb539680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:12:34.190 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:34.183+0000 7f50cb539680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:12:34.194 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:34.183+0000 7f50cb539680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T03:12:34.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 7 2023-01-25T03:12:34.451 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T03:12:34.452 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=7 2023-01-25T03:12:34.452 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T03:12:34.452 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T03:12:34.452 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:34.453 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T03:12:34.453 INFO:tasks.workunit.client.0.smithi008.stdout:0 2023-01-25T03:12:34.454 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:34.454 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:34.743 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:34.735+0000 7f50cb539680 -1 Falling back to public interface 2023-01-25T03:12:34.777 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:35.779 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:35.779 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:35.780 INFO:tasks.workunit.client.0.smithi008.stdout:1 2023-01-25T03:12:35.780 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T03:12:35.781 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:35.781 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:36.090 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:37.092 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:37.092 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:37.093 INFO:tasks.workunit.client.0.smithi008.stdout:2 2023-01-25T03:12:37.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T03:12:37.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:37.094 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:37.170 INFO:tasks.workunit.client.0.smithi008.stderr:2023-01-25T03:12:37.163+0000 7f50cb539680 -1 osd.7 0 log_to_monitors true 2023-01-25T03:12:37.393 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:38.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:38.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:38.396 INFO:tasks.workunit.client.0.smithi008.stdout:3 2023-01-25T03:12:38.396 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T03:12:38.397 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:38.397 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:38.702 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:39.703 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:39.704 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:39.704 INFO:tasks.workunit.client.0.smithi008.stdout:4 2023-01-25T03:12:39.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T03:12:39.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:39.705 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:40.095 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:41.097 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:41.098 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:41.098 INFO:tasks.workunit.client.0.smithi008.stdout:5 2023-01-25T03:12:41.098 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T03:12:41.099 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:41.099 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:41.412 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:42.413 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:42.414 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:42.414 INFO:tasks.workunit.client.0.smithi008.stdout:6 2023-01-25T03:12:42.415 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T03:12:42.415 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:42.415 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:42.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:43.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:43.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:43.737 INFO:tasks.workunit.client.0.smithi008.stdout:7 2023-01-25T03:12:43.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T03:12:43.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:43.738 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:44.226 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:45.228 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:45.229 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:45.230 INFO:tasks.workunit.client.0.smithi008.stdout:8 2023-01-25T03:12:45.231 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T03:12:45.231 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:45.232 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:45.542 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:46.544 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:46.545 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:46.545 INFO:tasks.workunit.client.0.smithi008.stdout:9 2023-01-25T03:12:46.546 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T03:12:46.547 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:46.547 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:46.868 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:47.869 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:47.870 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:47.870 INFO:tasks.workunit.client.0.smithi008.stdout:10 2023-01-25T03:12:47.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T03:12:47.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:47.871 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:48.298 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:49.300 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:49.301 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:49.301 INFO:tasks.workunit.client.0.smithi008.stdout:11 2023-01-25T03:12:49.302 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T03:12:49.302 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:49.302 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:49.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:50.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:50.689 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:50.690 INFO:tasks.workunit.client.0.smithi008.stdout:12 2023-01-25T03:12:50.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T03:12:50.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:50.691 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:51.409 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:52.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:52.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:52.411 INFO:tasks.workunit.client.0.smithi008.stdout:13 2023-01-25T03:12:52.412 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T03:12:52.412 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:52.413 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:53.037 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:54.040 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:54.040 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:54.040 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T03:12:54.041 INFO:tasks.workunit.client.0.smithi008.stdout:14 2023-01-25T03:12:54.041 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:54.041 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:54.856 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:55.858 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:55.858 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:55.858 INFO:tasks.workunit.client.0.smithi008.stdout:15 2023-01-25T03:12:55.859 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T03:12:55.859 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:55.859 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:56.280 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:57.283 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:57.283 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:57.284 INFO:tasks.workunit.client.0.smithi008.stdout:16 2023-01-25T03:12:57.285 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T03:12:57.285 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:57.285 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:57.733 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:12:58.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:12:58.735 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:12:58.736 INFO:tasks.workunit.client.0.smithi008.stdout:17 2023-01-25T03:12:58.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T03:12:58.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:12:58.737 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:12:59.107 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:13:00.109 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:13:00.109 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:13:00.110 INFO:tasks.workunit.client.0.smithi008.stdout:18 2023-01-25T03:13:00.110 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 18 2023-01-25T03:13:00.110 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:13:00.111 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:13:00.605 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T03:13:01.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T03:13:01.607 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T03:13:01.608 INFO:tasks.workunit.client.0.smithi008.stdout:19 2023-01-25T03:13:01.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 19 2023-01-25T03:13:01.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T03:13:01.609 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.7 up' 2023-01-25T03:13:01.991 INFO:tasks.workunit.client.0.smithi008.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/63222,v1:127.0.0.1:6859/63222] [v2:127.0.0.1:6860/63222,v1:127.0.0.1:6861/63222] exists,up b6f3d4ad-ad20-4a13-a640-fc38c165e64e 2023-01-25T03:13:01.992 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T03:13:01.992 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T03:13:01.993 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T03:13:01.993 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: create_pool test 32 32 2023-01-25T03:13:01.993 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2023-01-25T03:13:02.542 INFO:tasks.workunit.client.0.smithi008.stderr:pool 'test' created 2023-01-25T03:13:02.554 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T03:13:03.557 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: wait_for_clean 2023-01-25T03:13:03.557 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T03:13:03.557 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T03:13:03.558 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T03:13:03.558 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T03:13:03.558 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T03:13:03.559 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T03:13:03.559 INFO:tasks.workunit.client.0.smithi008.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T03:13:03.560 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T03:13:03.560 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T03:13:03.560 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T03:13:03.774 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T03:13:03.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T03:13:03.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T03:13:03.775 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T03:13:03.775 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T03:13:04.376 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T03:13:04.377 INFO:tasks.workunit.client.0.smithi008.stderr:1 2023-01-25T03:13:04.377 INFO:tasks.workunit.client.0.smithi008.stderr:2 2023-01-25T03:13:04.377 INFO:tasks.workunit.client.0.smithi008.stderr:3 2023-01-25T03:13:04.377 INFO:tasks.workunit.client.0.smithi008.stderr:4 2023-01-25T03:13:04.378 INFO:tasks.workunit.client.0.smithi008.stderr:5 2023-01-25T03:13:04.378 INFO:tasks.workunit.client.0.smithi008.stderr:6 2023-01-25T03:13:04.378 INFO:tasks.workunit.client.0.smithi008.stderr:7' 2023-01-25T03:13:04.378 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T03:13:04.378 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:04.379 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T03:13:04.794 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836543 2023-01-25T03:13:04.794 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836543 2023-01-25T03:13:04.795 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543' 2023-01-25T03:13:04.795 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:04.795 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T03:13:05.017 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673011 2023-01-25T03:13:05.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673011 2023-01-25T03:13:05.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011' 2023-01-25T03:13:05.018 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:05.019 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T03:13:05.170 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509486 2023-01-25T03:13:05.171 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509486 2023-01-25T03:13:05.171 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011 2-64424509486' 2023-01-25T03:13:05.171 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:05.172 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T03:13:05.315 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345956 2023-01-25T03:13:05.316 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345956 2023-01-25T03:13:05.316 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011 2-64424509486 3-85899345956' 2023-01-25T03:13:05.316 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:05.317 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-25T03:13:05.483 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182432 2023-01-25T03:13:05.484 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182432 2023-01-25T03:13:05.485 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011 2-64424509486 3-85899345956 4-107374182432' 2023-01-25T03:13:05.485 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:05.485 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2023-01-25T03:13:05.733 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=128849018903 2023-01-25T03:13:05.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 128849018903 2023-01-25T03:13:05.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011 2-64424509486 3-85899345956 4-107374182432 5-128849018903' 2023-01-25T03:13:05.734 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:05.735 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2023-01-25T03:13:05.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=150323855381 2023-01-25T03:13:05.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 150323855381 2023-01-25T03:13:05.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011 2-64424509486 3-85899345956 4-107374182432 5-128849018903 6-150323855381' 2023-01-25T03:13:05.901 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T03:13:05.902 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2023-01-25T03:13:06.092 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=171798691852 2023-01-25T03:13:06.092 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 171798691852 2023-01-25T03:13:06.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836543 1-42949673011 2-64424509486 3-85899345956 4-107374182432 5-128849018903 6-150323855381 7-171798691852' 2023-01-25T03:13:06.093 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:06.093 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836543 2023-01-25T03:13:06.094 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:06.095 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T03:13:06.096 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836543 2023-01-25T03:13:06.097 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:06.099 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836543 2023-01-25T03:13:06.099 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2023-01-25T03:13:06.099 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.0 seq 21474836543 2023-01-25T03:13:06.100 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T03:13:06.408 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836543 -lt 21474836543 2023-01-25T03:13:06.408 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:06.409 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673011 2023-01-25T03:13:06.409 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:06.411 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T03:13:06.412 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673011 2023-01-25T03:13:06.412 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:06.414 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673011 2023-01-25T03:13:06.414 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.1 seq 42949673011 2023-01-25T03:13:06.415 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2023-01-25T03:13:06.415 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T03:13:06.720 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673010 -lt 42949673011 2023-01-25T03:13:06.720 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-25T03:13:07.721 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-25T03:13:07.722 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T03:13:08.034 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673012 -lt 42949673011 2023-01-25T03:13:08.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:08.035 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509486 2023-01-25T03:13:08.036 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:08.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T03:13:08.039 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509486 2023-01-25T03:13:08.039 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:08.041 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509486 2023-01-25T03:13:08.041 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2023-01-25T03:13:08.042 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.2 seq 64424509486 2023-01-25T03:13:08.042 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T03:13:08.379 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509486 -lt 64424509486 2023-01-25T03:13:08.380 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:08.380 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345956 2023-01-25T03:13:08.381 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:08.382 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T03:13:08.383 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345956 2023-01-25T03:13:08.384 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:08.386 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345956 2023-01-25T03:13:08.386 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345956' 2023-01-25T03:13:08.386 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.3 seq 85899345956 2023-01-25T03:13:08.387 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T03:13:08.684 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345956 -lt 85899345956 2023-01-25T03:13:08.685 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:08.685 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182432 2023-01-25T03:13:08.686 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:08.687 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-25T03:13:08.689 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182432 2023-01-25T03:13:08.689 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:08.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182432 2023-01-25T03:13:08.690 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2023-01-25T03:13:08.691 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.4 seq 107374182432 2023-01-25T03:13:08.691 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-25T03:13:09.033 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182432 -lt 107374182432 2023-01-25T03:13:09.033 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:09.034 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 5-128849018903 2023-01-25T03:13:09.034 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:09.035 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=5 2023-01-25T03:13:09.036 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 5-128849018903 2023-01-25T03:13:09.037 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:09.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=128849018903 2023-01-25T03:13:09.038 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.5 seq 128849018903' 2023-01-25T03:13:09.038 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.5 seq 128849018903 2023-01-25T03:13:09.039 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 5 2023-01-25T03:13:09.395 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 128849018904 -lt 128849018903 2023-01-25T03:13:09.396 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:09.396 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 6-150323855381 2023-01-25T03:13:09.397 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:09.402 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=6 2023-01-25T03:13:09.402 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 6-150323855381 2023-01-25T03:13:09.402 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:09.403 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=150323855381 2023-01-25T03:13:09.403 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.6 seq 150323855381' 2023-01-25T03:13:09.403 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.6 seq 150323855381 2023-01-25T03:13:09.403 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 6 2023-01-25T03:13:09.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 150323855382 -lt 150323855381 2023-01-25T03:13:09.736 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T03:13:09.737 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 7-171798691852 2023-01-25T03:13:09.737 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T03:13:09.739 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=7 2023-01-25T03:13:09.740 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 7-171798691852 2023-01-25T03:13:09.740 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T03:13:09.741 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=171798691852 2023-01-25T03:13:09.742 INFO:tasks.workunit.client.0.smithi008.stdout:waiting osd.7 seq 171798691852 2023-01-25T03:13:09.742 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.7 seq 171798691852' 2023-01-25T03:13:09.743 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 7 2023-01-25T03:13:10.046 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 171798691853 -lt 171798691852 2023-01-25T03:13:10.047 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T03:13:10.047 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T03:13:10.048 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T03:13:10.487 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 32 == 0 2023-01-25T03:13:10.488 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T03:13:10.488 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T03:13:10.489 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T03:13:10.489 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T03:13:10.489 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T03:13:10.490 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T03:13:10.490 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T03:13:10.830 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=32 2023-01-25T03:13:10.831 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T03:13:10.831 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T03:13:10.831 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T03:13:11.198 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 32 = 32 2023-01-25T03:13:11.198 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T03:13:11.198 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T03:13:11.199 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: ceph osd dump 2023-01-25T03:13:11.199 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2023-01-25T03:13:11.199 INFO:tasks.workunit.client.0.smithi008.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: awk '{ print $2 }' 2023-01-25T03:13:11.545 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: poolid=1 2023-01-25T03:13:11.546 INFO:tasks.workunit.client.0.smithi008.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:252: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.39517 bs=1M count=50 2023-01-25T03:13:11.692 INFO:tasks.workunit.client.0.smithi008.stderr:50+0 records in 2023-01-25T03:13:11.693 INFO:tasks.wo