2023-01-24T18:18:52.313 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T18:18:52.314 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_16:05:13-rados-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135581 --verbose --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} --owner scheduled_yuriw@teuthology --name yuriw-2023-01-24_16:05:13-rados-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-01-24_16:05:13-rados-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135581/orig.config.yaml 2023-01-24T18:18:52.338 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T18:18:52.394 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_16:05:13-rados-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135581 branch: wip-yuri7-testing-2023-01-23-1532-quincy description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} email: yweinstein@radhat.com first_in_suite: false job_id: '7135581' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_16:05:13-rados-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri7-testing-2023-01-23-1532-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 49f8fb05584886826e8eade75f7105fba754560c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 49f8fb05584886826e8eade75f7105fba754560c workunit: branch: wip-yuri7-testing-2023-01-23-1532-quincy sha1: 49f8fb05584886826e8eade75f7105fba754560c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 49f8fb05584886826e8eade75f7105fba754560c sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri7-testing-2023-01-23-1532-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49f8fb05584886826e8eade75f7105fba754560c targets: smithi082.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd-backfill teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_16:05:13 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T18:18:52.394 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa; will attempt to use it 2023-01-24T18:18:52.395 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks 2023-01-24T18:18:52.395 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T18:18:52.397 INFO:teuthology.task.internal:Checking packages... 2023-01-24T18:18:52.425 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '49f8fb05584886826e8eade75f7105fba754560c' 2023-01-24T18:18:52.426 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T18:18:52.426 INFO:teuthology.packaging:ref: None 2023-01-24T18:18:52.427 INFO:teuthology.packaging:tag: None 2023-01-24T18:18:52.427 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T18:18:52.427 INFO:teuthology.packaging:sha1: 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T18:18:52.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T18:18:52.585 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-727-g49f8fb05-1focal 2023-01-24T18:18:52.585 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T18:18:52.592 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T18:18:52.592 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T18:18:52.628 INFO:teuthology.task.internal:Saving configuration 2023-01-24T18:18:52.638 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T18:18:52.654 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T18:18:52.682 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_16:05:13-rados-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135581', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 18:13:05.082836', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-01-24T18:18:52.682 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T18:18:52.693 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-24T18:18:52.693 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T18:18:52.748 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcbb1bf1700>, signals=[15]) 2023-01-24T18:18:52.749 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T18:18:52.756 INFO:teuthology.task.internal:Opening connections... 2023-01-24T18:18:52.756 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2023-01-24T18:18:52.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T18:18:52.811 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T18:18:52.819 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2023-01-24T18:18:52.939 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2023-01-24T18:18:52.939 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2023-01-24T18:18:52.945 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Ubuntu" 2023-01-24T18:18:52.945 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-24T18:18:52.945 INFO:teuthology.orchestra.run.smithi082.stdout:ID=ubuntu 2023-01-24T18:18:52.946 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE=debian 2023-01-24T18:18:52.946 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-24T18:18:52.946 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="20.04" 2023-01-24T18:18:52.946 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-24T18:18:52.947 INFO:teuthology.orchestra.run.smithi082.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-24T18:18:52.947 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-24T18:18:52.947 INFO:teuthology.orchestra.run.smithi082.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-24T18:18:52.947 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_CODENAME=focal 2023-01-24T18:18:52.947 INFO:teuthology.orchestra.run.smithi082.stdout:UBUNTU_CODENAME=focal 2023-01-24T18:18:52.948 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2023-01-24T18:18:52.965 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T18:18:52.974 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T18:18:52.997 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T18:18:52.997 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2023-01-24T18:18:53.003 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T18:18:53.024 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T18:18:53.024 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2023-01-24T18:18:53.056 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T18:18:53.088 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T18:18:53.110 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T18:18:53.111 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T18:18:53.112 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T18:18:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T18:18:53.119 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-01-24T18:18:53.126 INFO:teuthology.orchestra.run.smithi082.stdout:5.4.0-124-generic 2023-01-24T18:18:53.126 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.4.0-124-generic 2023-01-24T18:18:53.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2023-01-24T18:18:53.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2023-01-24T18:18:53.639 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-24T18:18:53.640 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-01-24T18:18:53.695 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-01-24T18:18:54.033 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-01-24T18:18:54.064 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,969 kB] 2023-01-24T18:18:54.253 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,336 kB] 2023-01-24T18:18:54.257 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [543 kB] 2023-01-24T18:18:54.280 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [320 kB] 2023-01-24T18:18:54.303 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.7 kB] 2023-01-24T18:18:54.321 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,463 kB] 2023-01-24T18:18:54.356 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-01-24T18:18:54.373 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [207 kB] 2023-01-24T18:18:54.393 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [624 B] 2023-01-24T18:18:54.411 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [787 kB] 2023-01-24T18:18:54.439 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [576 kB] 2023-01-24T18:18:54.462 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [152 kB] 2023-01-24T18:18:54.481 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.9 kB] 2023-01-24T18:18:54.683 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [775 kB] 2023-01-24T18:18:54.702 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [403 kB] 2023-01-24T18:18:54.710 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-01-24T18:18:54.711 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.4 kB] 2023-01-24T18:18:54.712 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,560 kB] 2023-01-24T18:18:54.772 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [220 kB] 2023-01-24T18:18:54.777 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [620 B] 2023-01-24T18:18:54.777 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [708 kB] 2023-01-24T18:18:54.793 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,018 kB] 2023-01-24T18:18:54.816 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [236 kB] 2023-01-24T18:18:54.821 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2023-01-24T18:18:54.858 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.2 kB] 2023-01-24T18:18:54.859 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-01-24T18:18:54.859 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [604 B] 2023-01-24T18:18:54.860 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-01-24T18:18:54.860 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB] 2023-01-24T18:18:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-01-24T18:18:57.834 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 13.8 MB in 2s (5,697 kB/s) 2023-01-24T18:18:59.144 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2023-01-24T18:18:59.168 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-24T18:18:59.233 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2023-01-24T18:18:59.397 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2023-01-24T18:18:59.397 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2023-01-24T18:18:59.552 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2023-01-24T18:18:59.552 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-24T18:18:59.552 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-generic linux-image-5.4.0-137-generic 2023-01-24T18:18:59.553 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-5.4.0-137-generic linux-modules-extra-5.4.0-137-generic 2023-01-24T18:18:59.554 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2023-01-24T18:18:59.554 INFO:teuthology.orchestra.run.smithi082.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-01-24T18:18:59.554 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2023-01-24T18:18:59.554 INFO:teuthology.orchestra.run.smithi082.stdout: thermald 2023-01-24T18:18:59.614 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2023-01-24T18:18:59.616 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-24T18:18:59.616 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.4.0-137-generic linux-modules-5.4.0-137-generic 2023-01-24T18:18:59.617 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-extra-5.4.0-137-generic 2023-01-24T18:18:59.617 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages will be upgraded: 2023-01-24T18:18:59.617 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-generic linux-image-generic 2023-01-24T18:18:59.740 INFO:teuthology.orchestra.run.smithi082.stdout:3 upgraded, 5 newly installed, 0 to remove and 125 not upgraded. 2023-01-24T18:18:59.740 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 77.1 MB of archives. 2023-01-24T18:18:59.740 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 380 MB of additional disk space will be used. 2023-01-24T18:18:59.741 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-137-generic amd64 5.4.0-137.154 [15.0 MB] 2023-01-24T18:19:00.035 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-137-generic amd64 5.4.0-137.154 [10.5 MB] 2023-01-24T18:19:00.124 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-137-generic amd64 5.4.0-137.154 [39.2 MB] 2023-01-24T18:19:00.591 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.137.135 [1,896 B] 2023-01-24T18:19:00.591 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.137.135 [2,528 B] 2023-01-24T18:19:00.591 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137 all 5.4.0-137.154 [11.0 MB] 2023-01-24T18:19:00.683 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137-generic amd64 5.4.0-137.154 [1,400 kB] 2023-01-24T18:19:00.694 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.137.135 [2,392 B] 2023-01-24T18:19:01.394 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 77.1 MB in 1s (71.7 MB/s) 2023-01-24T18:19:01.573 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-5.4.0-137-generic. 2023-01-24T18:19:03.309 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-01-24T18:19:03.315 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../0-linux-modules-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T18:19:03.403 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:05.469 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-image-5.4.0-137-generic. 2023-01-24T18:19:05.485 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../1-linux-image-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T18:19:05.596 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:06.697 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-137-generic. 2023-01-24T18:19:06.714 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T18:19:06.755 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:11.504 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../3-linux-generic_5.4.0.137.135_amd64.deb ... 2023-01-24T18:19:11.629 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-24T18:19:11.974 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.137.135_amd64.deb ... 2023-01-24T18:19:12.099 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-24T18:19:12.427 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.4.0-137. 2023-01-24T18:19:12.442 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../5-linux-headers-5.4.0-137_5.4.0-137.154_all.deb ... 2023-01-24T18:19:12.485 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-24T18:19:17.813 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.4.0-137-generic. 2023-01-24T18:19:17.836 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../6-linux-headers-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T18:19:17.879 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:20.495 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.137.135_amd64.deb ... 2023-01-24T18:19:20.617 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-24T18:19:20.912 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-24T18:19:21.054 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:22.289 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:22.415 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:23.555 INFO:teuthology.orchestra.run.smithi082.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-137-generic 2023-01-24T18:19:23.555 INFO:teuthology.orchestra.run.smithi082.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-137-generic 2023-01-24T18:19:23.683 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:24.851 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-generic (5.4.0.137.135) ... 2023-01-24T18:19:24.977 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-generic (5.4.0.137.135) ... 2023-01-24T18:19:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-generic (5.4.0.137.135) ... 2023-01-24T18:19:25.665 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T18:19:26.094 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-01-24T18:19:26.095 INFO:teuthology.orchestra.run.smithi082.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-137-generic 2023-01-24T18:19:48.123 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-01-24T18:19:48.124 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub' 2023-01-24T18:19:48.135 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-24T18:19:48.148 INFO:teuthology.orchestra.run.smithi082.stdout:Generating grub configuration file ... 2023-01-24T18:19:48.467 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-24T18:19:48.513 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-24T18:19:48.910 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-24T18:19:48.921 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-24T18:19:51.668 INFO:teuthology.orchestra.run.smithi082.stdout:done 2023-01-24T18:19:52.126 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2023-01-24T18:19:52.180 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2023-01-24T18:19:52.180 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2023-01-24T18:19:52.180 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2023-01-24T18:19:52.180 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2023-01-24T18:19:52.181 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 19 2023-01-24T18:19:52.181 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2023-01-24T18:19:52.181 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2023-01-24T18:19:52.181 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2023-01-24T18:19:52.181 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.4.0.137.135 2023-01-24T18:19:52.182 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-24T18:19:52.182 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-24T18:19:52.182 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2023-01-24T18:19:52.182 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2023-01-24T18:19:52.182 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2023-01-24T18:19:52.183 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2023-01-24T18:19:52.183 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-137-generic 2023-01-24T18:19:52.183 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T18:19:52.183 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T18:19:52.184 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-24T18:19:52.184 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T18:19:52.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2023-01-24T18:19:52.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2023-01-24T18:19:52.392 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-24T18:19:52.398 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-24T18:19:52.415 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-24T18:19:52.508 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-24T18:19:54.024 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2023-01-24T18:19:54.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-24T18:19:54.118 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2023-01-24T18:19:54.306 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2023-01-24T18:19:54.306 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2023-01-24T18:19:54.506 INFO:teuthology.orchestra.run.smithi082.stdout:linux-image-generic is already the newest version (5.4.0.137.135). 2023-01-24T18:19:54.507 INFO:teuthology.orchestra.run.smithi082.stdout:0 upgraded, 0 newly installed, 0 to remove and 125 not upgraded. 2023-01-24T18:19:54.508 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2023-01-24T18:19:54.528 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2023-01-24T18:19:54.529 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2023-01-24T18:19:54.529 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2023-01-24T18:19:54.529 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2023-01-24T18:19:54.529 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 19 2023-01-24T18:19:54.530 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2023-01-24T18:19:54.530 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2023-01-24T18:19:54.530 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2023-01-24T18:19:54.530 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.4.0.137.135 2023-01-24T18:19:54.530 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-24T18:19:54.531 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-24T18:19:54.531 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2023-01-24T18:19:54.531 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2023-01-24T18:19:54.531 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2023-01-24T18:19:54.531 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2023-01-24T18:19:54.532 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2023-01-24T18:19:54.582 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.15giuaJ41i 2023-01-24T18:19:54.584 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub/grub.cfg /tmp/tmp.15giuaJ41i 2023-01-24T18:19:54.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.15giuaJ41i 2023-01-24T18:19:54.692 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.15giuaJ41i is 9KB 2023-01-24T18:19:54.706 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.15giuaJ41i 2023-01-24T18:19:54.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-01-24T18:19:54.768 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:19:54.768 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-01-24T18:19:54.769 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-01-24T18:19:54.804 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-137-generic 2023-01-24T18:19:54.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo update-grub 2023-01-24T18:19:56.158 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub' 2023-01-24T18:19:56.164 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-24T18:19:56.175 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2023-01-24T18:19:56.412 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-24T18:19:56.424 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-24T18:19:56.776 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-24T18:19:56.787 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-24T18:19:58.401 INFO:teuthology.orchestra.run.smithi082.stderr:done 2023-01-24T18:19:58.403 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2023-01-24T18:20:28.434 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-24T18:20:28.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-01-24T18:20:28.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T18:20:46.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2023-01-24T18:21:16.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-01-24T18:21:16.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T18:21:35.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2023-01-24T18:22:05.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-01-24T18:22:05.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T18:22:05.459 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-01-24T18:22:06.039 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-01-24T18:22:06.040 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-137-generic"... 2023-01-24T18:22:06.041 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-01-24T18:22:06.047 INFO:teuthology.orchestra.run.smithi082.stdout:5.4.0-137-generic 2023-01-24T18:22:06.048 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-137-generic vs 5.4.0-137-generic 2023-01-24T18:22:06.048 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T18:22:06.049 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-01-24T18:22:07.050 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T18:22:07.050 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T18:22:07.150 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2023-01-24T18:22:07.174 DEBUG:teuthology.parallel:result is None 2023-01-24T18:22:07.174 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T18:22:07.181 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T18:22:07.182 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T18:22:07.189 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T18:22:07.219 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T18:22:07.256 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T18:22:07.256 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T18:22:07.292 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T18:22:07.302 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T18:22:07.303 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T18:22:07.323 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:22:07.334 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:22:07.336 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T18:22:07.369 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T18:22:07.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T18:22:07.397 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T18:22:07.409 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T18:22:07.410 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T18:22:07.416 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T18:22:07.466 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T18:22:07.473 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:22:07.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T18:22:07.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2023-01-24T18:22:07.541 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T18:22:07.549 INFO:teuthology.task.internal:Starting timer... 2023-01-24T18:22:07.550 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T18:22:07.572 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T18:22:07.607 DEBUG:teuthology.task.selinux:Excluding smithi082: OS 'ubuntu' does not support SELinux 2023-01-24T18:22:07.607 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T18:22:07.607 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-01-24T18:22:07.608 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T18:22:07.608 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T18:22:07.638 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-24T18:22:07.643 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-24T18:22:07.771 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T18:22:07.788 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T18:22:07.789 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T18:26:49.150 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2023-01-24T18:26:49.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-01-24T18:26:49.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T18:26:49.199 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-01-24T18:26:49.322 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T18:26:49.334 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T18:26:49.335 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T18:26:49.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T18:26:49.367 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-01-24T18:26:49.367 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Command line: ntpd -gq 2023-01-24T18:26:49.368 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: proto: precision = 0.070 usec (-24) 2023-01-24T18:26:49.369 INFO:teuthology.orchestra.run.smithi082.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-24T18:26:49.369 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-24T18:26:49.371 INFO:teuthology.orchestra.run.smithi082.stderr:restrict ::: KOD does nothing without LIMITED. 2023-01-24T18:26:49.373 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: restrict ::: KOD does nothing without LIMITED. 2023-01-24T18:26:49.374 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listen and drop on 0 v6wildcard [::]:123 2023-01-24T18:26:49.374 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-01-24T18:26:49.374 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listen normally on 2 lo 127.0.0.1:123 2023-01-24T18:26:49.375 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listen normally on 3 enp3s0f1 172.21.15.82:123 2023-01-24T18:26:49.375 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listen normally on 4 lo [::1]:123 2023-01-24T18:26:49.376 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3ff%5]:123 2023-01-24T18:26:49.376 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:49 ntpd[19417]: Listening on routing socket on fd #22 for interface updates 2023-01-24T18:26:53.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:53 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:53.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:53 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:53.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:53 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:53.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:53 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:53.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:53 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:53.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:53 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:54.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:54.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:54.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:54.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:54.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:54.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:54.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:54.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:54.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:54.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:54.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:54.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:54.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:54.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:54 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:55.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:55 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:55.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:55 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:55.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:55 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:55.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:55 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:55.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:55 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:55.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:55 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:56.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:56.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:56.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:56.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:56.372 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:56.372 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:56.372 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:56.372 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:56.372 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:56.373 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:56.373 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:56.373 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:56 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:57.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:57 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:57.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:57 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:57.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:57 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:57.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:57 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:57.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:57 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:57.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:57 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:58.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:58.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:58.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:58.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:58.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:58.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:58.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:58.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:58.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:58.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:58.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:58.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:58 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:59.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:59.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:59.368 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T18:26:59.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T18:26:59.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T18:26:59.369 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T18:26:59.370 INFO:teuthology.orchestra.run.smithi082.stdout:24 Jan 18:26:59 ntpd[19417]: ntpd: time slew -0.003253 s 2023-01-24T18:26:59.370 INFO:teuthology.orchestra.run.smithi082.stdout:ntpd: time slew -0.003253s 2023-01-24T18:26:59.370 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-01-24T18:26:59.371 INFO:teuthology.orchestra.run.smithi082.stderr:24 Jan 18:26:59 ntpd[19417]: can't open /var/log/ntpstats/loopstats.20230124: Permission denied 2023-01-24T18:26:59.441 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2023-01-24T18:26:59.441 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2023-01-24T18:26:59.442 INFO:teuthology.orchestra.run.smithi082.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T18:26:59.442 INFO:teuthology.orchestra.run.smithi082.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T18:26:59.442 INFO:teuthology.orchestra.run.smithi082.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T18:26:59.442 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T18:26:59.443 INFO:teuthology.run_tasks:Running task install... 2023-01-24T18:26:59.453 DEBUG:teuthology.task.install:project ceph 2023-01-24T18:26:59.453 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}} 2023-01-24T18:26:59.453 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'} 2023-01-24T18:26:59.454 INFO:teuthology.task.install:Using flavor: default 2023-01-24T18:26:59.459 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T18:26:59.460 INFO:teuthology.task.install:extra packages: [] 2023-01-24T18:26:59.460 DEBUG:teuthology.orchestra.run.smithi082:> sudo lsb_release -is 2023-01-24T18:26:59.533 INFO:teuthology.orchestra.run.smithi082.stdout:Ubuntu 2023-01-24T18:26:59.538 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-24T18:26:59.539 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-key list | grep Ceph 2023-01-24T18:26:59.635 INFO:teuthology.orchestra.run.smithi082.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-01-24T18:26:59.697 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-01-24T18:26:59.698 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph.com (release key) 2023-01-24T18:26:59.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo lsb_release -is 2023-01-24T18:26:59.748 INFO:teuthology.orchestra.run.smithi082.stdout:Ubuntu 2023-01-24T18:26:59.753 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-24T18:26:59.753 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-01-24T18:26:59.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T18:26:59.903 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default/ 2023-01-24T18:26:59.904 INFO:teuthology.task.install.deb:Package version is 17.2.5-727-g49f8fb05-1focal 2023-01-24T18:27:00.023 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:27:00.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-01-24T18:27:00.040 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2023-01-24T18:27:00.293 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal InRelease 2023-01-24T18:27:00.318 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-01-24T18:27:00.344 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-24T18:27:00.345 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-24T18:27:00.346 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal Release.gpg 2023-01-24T18:27:00.371 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main i386 Packages [3,372 B] 2023-01-24T18:27:00.409 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-01-24T18:27:00.412 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-24T18:27:00.492 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-24T18:27:00.984 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 28.4 kB in 1s (38.2 kB/s) 2023-01-24T18:27:01.925 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2023-01-24T18:27:01.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.5-727-g49f8fb05-1focal cephadm=17.2.5-727-g49f8fb05-1focal ceph-mds=17.2.5-727-g49f8fb05-1focal ceph-mgr=17.2.5-727-g49f8fb05-1focal ceph-common=17.2.5-727-g49f8fb05-1focal ceph-fuse=17.2.5-727-g49f8fb05-1focal ceph-test=17.2.5-727-g49f8fb05-1focal ceph-volume=17.2.5-727-g49f8fb05-1focal radosgw=17.2.5-727-g49f8fb05-1focal python3-rados=17.2.5-727-g49f8fb05-1focal python3-rgw=17.2.5-727-g49f8fb05-1focal python3-cephfs=17.2.5-727-g49f8fb05-1focal python3-rbd=17.2.5-727-g49f8fb05-1focal libcephfs2=17.2.5-727-g49f8fb05-1focal libcephfs-dev=17.2.5-727-g49f8fb05-1focal librados2=17.2.5-727-g49f8fb05-1focal librbd1=17.2.5-727-g49f8fb05-1focal rbd-fuse=17.2.5-727-g49f8fb05-1focal 2023-01-24T18:27:02.020 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2023-01-24T18:27:02.206 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2023-01-24T18:27:02.206 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2023-01-24T18:27:02.370 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T18:27:02.371 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T18:27:02.371 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T18:27:02.371 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2023-01-24T18:27:02.372 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-01-24T18:27:02.372 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-01-24T18:27:02.372 INFO:teuthology.orchestra.run.smithi082.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-01-24T18:27:02.372 INFO:teuthology.orchestra.run.smithi082.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-01-24T18:27:02.372 INFO:teuthology.orchestra.run.smithi082.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-01-24T18:27:02.373 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-01-24T18:27:02.373 INFO:teuthology.orchestra.run.smithi082.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-01-24T18:27:02.374 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-01-24T18:27:02.374 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-01-24T18:27:02.374 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-01-24T18:27:02.374 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-01-24T18:27:02.374 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-01-24T18:27:02.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-01-24T18:27:02.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-01-24T18:27:02.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-01-24T18:27:02.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-01-24T18:27:02.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-01-24T18:27:02.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-01-24T18:27:02.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-01-24T18:27:02.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-01-24T18:27:02.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-01-24T18:27:02.376 INFO:teuthology.orchestra.run.smithi082.stdout: socat xmlstarlet 2023-01-24T18:27:02.377 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2023-01-24T18:27:02.377 INFO:teuthology.orchestra.run.smithi082.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-01-24T18:27:02.377 INFO:teuthology.orchestra.run.smithi082.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-01-24T18:27:02.377 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-01-24T18:27:02.377 INFO:teuthology.orchestra.run.smithi082.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-01-24T18:27:02.378 INFO:teuthology.orchestra.run.smithi082.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-01-24T18:27:02.378 INFO:teuthology.orchestra.run.smithi082.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-01-24T18:27:02.378 INFO:teuthology.orchestra.run.smithi082.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-01-24T18:27:02.378 INFO:teuthology.orchestra.run.smithi082.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-01-24T18:27:02.378 INFO:teuthology.orchestra.run.smithi082.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-01-24T18:27:02.379 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2023-01-24T18:27:02.379 INFO:teuthology.orchestra.run.smithi082.stdout: btrfs-tools 2023-01-24T18:27:02.490 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2023-01-24T18:27:02.490 INFO:teuthology.orchestra.run.smithi082.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-01-24T18:27:02.490 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-01-24T18:27:02.491 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-01-24T18:27:02.491 INFO:teuthology.orchestra.run.smithi082.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-01-24T18:27:02.491 INFO:teuthology.orchestra.run.smithi082.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-01-24T18:27:02.491 INFO:teuthology.orchestra.run.smithi082.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-01-24T18:27:02.491 INFO:teuthology.orchestra.run.smithi082.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-01-24T18:27:02.492 INFO:teuthology.orchestra.run.smithi082.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-01-24T18:27:02.493 INFO:teuthology.orchestra.run.smithi082.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-01-24T18:27:02.493 INFO:teuthology.orchestra.run.smithi082.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-01-24T18:27:02.493 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-01-24T18:27:02.494 INFO:teuthology.orchestra.run.smithi082.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-01-24T18:27:02.494 INFO:teuthology.orchestra.run.smithi082.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-01-24T18:27:02.494 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-01-24T18:27:02.494 INFO:teuthology.orchestra.run.smithi082.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-01-24T18:27:02.494 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-01-24T18:27:02.495 INFO:teuthology.orchestra.run.smithi082.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-01-24T18:27:02.495 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-01-24T18:27:02.495 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-01-24T18:27:02.495 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-01-24T18:27:02.495 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-01-24T18:27:02.496 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-01-24T18:27:02.496 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-01-24T18:27:02.496 INFO:teuthology.orchestra.run.smithi082.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-01-24T18:27:02.496 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages will be upgraded: 2023-01-24T18:27:02.496 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 librbd1 2023-01-24T18:27:02.674 INFO:teuthology.orchestra.run.smithi082.stdout:2 upgraded, 108 newly installed, 0 to remove and 120 not upgraded. 2023-01-24T18:27:02.674 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 209 MB of archives. 2023-01-24T18:27:02.674 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 980 MB of additional disk space will be used. 2023-01-24T18:27:02.675 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.5-727-g49f8fb05-1focal [3,502 kB] 2023-01-24T18:27:03.130 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.5-727-g49f8fb05-1focal [3,550 kB] 2023-01-24T18:27:03.337 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.5-727-g49f8fb05-1focal [755 kB] 2023-01-24T18:27:03.490 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-01-24T18:27:03.512 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.5-727-g49f8fb05-1focal [412 kB] 2023-01-24T18:27:03.520 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.5-727-g49f8fb05-1focal [22.5 kB] 2023-01-24T18:27:03.560 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.5-727-g49f8fb05-1focal [200 kB] 2023-01-24T18:27:03.565 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.5-727-g49f8fb05-1focal [43.4 kB] 2023-01-24T18:27:03.566 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.5-727-g49f8fb05-1focal [384 kB] 2023-01-24T18:27:03.573 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-01-24T18:27:03.575 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-01-24T18:27:03.584 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-01-24T18:27:03.586 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.5-727-g49f8fb05-1focal [5,528 kB] 2023-01-24T18:27:03.591 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-01-24T18:27:03.617 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-01-24T18:27:03.617 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-01-24T18:27:03.625 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-01-24T18:27:03.656 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-01-24T18:27:03.658 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-01-24T18:27:03.664 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-01-24T18:27:03.666 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-01-24T18:27:03.666 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-01-24T18:27:03.669 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-01-24T18:27:03.669 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-01-24T18:27:03.670 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-01-24T18:27:03.671 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-01-24T18:27:03.674 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-01-24T18:27:03.689 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-01-24T18:27:03.689 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-01-24T18:27:03.690 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-01-24T18:27:03.690 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-01-24T18:27:03.691 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-01-24T18:27:03.691 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-01-24T18:27:03.693 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-01-24T18:27:03.693 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-01-24T18:27:03.695 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-01-24T18:27:03.707 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-01-24T18:27:03.708 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-01-24T18:27:03.725 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-01-24T18:27:03.725 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-01-24T18:27:03.726 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-01-24T18:27:03.726 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-01-24T18:27:03.727 INFO:teuthology.orchestra.run.smithi082.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2023-01-24T18:27:03.729 INFO:teuthology.orchestra.run.smithi082.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-01-24T18:27:03.730 INFO:teuthology.orchestra.run.smithi082.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-01-24T18:27:03.730 INFO:teuthology.orchestra.run.smithi082.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-01-24T18:27:03.742 INFO:teuthology.orchestra.run.smithi082.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-01-24T18:27:03.743 INFO:teuthology.orchestra.run.smithi082.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-01-24T18:27:03.759 INFO:teuthology.orchestra.run.smithi082.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-01-24T18:27:03.760 INFO:teuthology.orchestra.run.smithi082.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-01-24T18:27:03.760 INFO:teuthology.orchestra.run.smithi082.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-01-24T18:27:03.787 INFO:teuthology.orchestra.run.smithi082.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.5-727-g49f8fb05-1focal [116 kB] 2023-01-24T18:27:03.789 INFO:teuthology.orchestra.run.smithi082.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.5-727-g49f8fb05-1focal [479 kB] 2023-01-24T18:27:03.823 INFO:teuthology.orchestra.run.smithi082.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.5-727-g49f8fb05-1focal [27.1 MB] 2023-01-24T18:27:03.841 INFO:teuthology.orchestra.run.smithi082.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-01-24T18:27:03.857 INFO:teuthology.orchestra.run.smithi082.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-01-24T18:27:03.858 INFO:teuthology.orchestra.run.smithi082.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-01-24T18:27:03.925 INFO:teuthology.orchestra.run.smithi082.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-01-24T18:27:03.926 INFO:teuthology.orchestra.run.smithi082.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-01-24T18:27:03.927 INFO:teuthology.orchestra.run.smithi082.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-01-24T18:27:03.928 INFO:teuthology.orchestra.run.smithi082.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-01-24T18:27:03.929 INFO:teuthology.orchestra.run.smithi082.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-01-24T18:27:03.930 INFO:teuthology.orchestra.run.smithi082.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-01-24T18:27:03.931 INFO:teuthology.orchestra.run.smithi082.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-01-24T18:27:03.934 INFO:teuthology.orchestra.run.smithi082.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-01-24T18:27:03.934 INFO:teuthology.orchestra.run.smithi082.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-01-24T18:27:03.943 INFO:teuthology.orchestra.run.smithi082.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-01-24T18:27:03.945 INFO:teuthology.orchestra.run.smithi082.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-01-24T18:27:03.961 INFO:teuthology.orchestra.run.smithi082.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-01-24T18:27:03.963 INFO:teuthology.orchestra.run.smithi082.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-01-24T18:27:03.964 INFO:teuthology.orchestra.run.smithi082.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-01-24T18:27:03.968 INFO:teuthology.orchestra.run.smithi082.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-01-24T18:27:03.972 INFO:teuthology.orchestra.run.smithi082.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-01-24T18:27:03.986 INFO:teuthology.orchestra.run.smithi082.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-01-24T18:27:03.989 INFO:teuthology.orchestra.run.smithi082.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-01-24T18:27:04.028 INFO:teuthology.orchestra.run.smithi082.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-01-24T18:27:04.029 INFO:teuthology.orchestra.run.smithi082.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-01-24T18:27:04.030 INFO:teuthology.orchestra.run.smithi082.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-01-24T18:27:04.030 INFO:teuthology.orchestra.run.smithi082.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-01-24T18:27:04.030 INFO:teuthology.orchestra.run.smithi082.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-01-24T18:27:04.031 INFO:teuthology.orchestra.run.smithi082.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-01-24T18:27:04.043 INFO:teuthology.orchestra.run.smithi082.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-01-24T18:27:04.043 INFO:teuthology.orchestra.run.smithi082.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-01-24T18:27:04.077 INFO:teuthology.orchestra.run.smithi082.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-01-24T18:27:04.078 INFO:teuthology.orchestra.run.smithi082.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-01-24T18:27:04.080 INFO:teuthology.orchestra.run.smithi082.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-01-24T18:27:04.082 INFO:teuthology.orchestra.run.smithi082.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-01-24T18:27:04.083 INFO:teuthology.orchestra.run.smithi082.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-01-24T18:27:04.084 INFO:teuthology.orchestra.run.smithi082.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-01-24T18:27:04.087 INFO:teuthology.orchestra.run.smithi082.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-01-24T18:27:04.095 INFO:teuthology.orchestra.run.smithi082.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-01-24T18:27:04.896 INFO:teuthology.orchestra.run.smithi082.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.5-727-g49f8fb05-1focal [6,439 kB] 2023-01-24T18:27:05.093 INFO:teuthology.orchestra.run.smithi082.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.5-727-g49f8fb05-1focal [218 kB] 2023-01-24T18:27:05.096 INFO:teuthology.orchestra.run.smithi082.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.5-727-g49f8fb05-1focal [152 kB] 2023-01-24T18:27:05.190 INFO:teuthology.orchestra.run.smithi082.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.5-727-g49f8fb05-1focal [1,583 kB] 2023-01-24T18:27:05.232 INFO:teuthology.orchestra.run.smithi082.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.5-727-g49f8fb05-1focal [7,828 kB] 2023-01-24T18:27:05.515 INFO:teuthology.orchestra.run.smithi082.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.5-727-g49f8fb05-1focal [28.6 MB] 2023-01-24T18:27:06.603 INFO:teuthology.orchestra.run.smithi082.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.5-727-g49f8fb05-1focal [3,904 B] 2023-01-24T18:27:06.604 INFO:teuthology.orchestra.run.smithi082.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.5-727-g49f8fb05-1focal [865 kB] 2023-01-24T18:27:06.618 INFO:teuthology.orchestra.run.smithi082.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.5-727-g49f8fb05-1focal [2,387 kB] 2023-01-24T18:27:06.635 INFO:teuthology.orchestra.run.smithi082.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.5-727-g49f8fb05-1focal [77.0 kB] 2023-01-24T18:27:06.636 INFO:teuthology.orchestra.run.smithi082.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.5-727-g49f8fb05-1focal [111 kB] 2023-01-24T18:27:06.636 INFO:teuthology.orchestra.run.smithi082.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.5-727-g49f8fb05-1focal [1,713 kB] 2023-01-24T18:27:06.654 INFO:teuthology.orchestra.run.smithi082.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.5-727-g49f8fb05-1focal [7,878 kB] 2023-01-24T18:27:06.914 INFO:teuthology.orchestra.run.smithi082.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.5-727-g49f8fb05-1focal [4,020 B] 2023-01-24T18:27:06.915 INFO:teuthology.orchestra.run.smithi082.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.5-727-g49f8fb05-1focal [56.6 MB] 2023-01-24T18:27:09.026 INFO:teuthology.orchestra.run.smithi082.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.5-727-g49f8fb05-1focal [225 kB] 2023-01-24T18:27:09.104 INFO:teuthology.orchestra.run.smithi082.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.5-727-g49f8fb05-1focal [21.8 kB] 2023-01-24T18:27:09.104 INFO:teuthology.orchestra.run.smithi082.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.5-727-g49f8fb05-1focal [16.9 MB] 2023-01-24T18:27:09.641 INFO:teuthology.orchestra.run.smithi082.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.5-727-g49f8fb05-1focal [80.1 kB] 2023-01-24T18:27:10.261 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 209 MB in 7s (29.2 MB/s) 2023-01-24T18:27:10.422 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-01-24T18:27:10.479 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141754 files and directories currently installed.) 2023-01-24T18:27:10.484 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-01-24T18:27:10.570 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-24T18:27:10.902 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../001-librbd1_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:11.040 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librbd1 (17.2.5-727-g49f8fb05-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-24T18:27:11.736 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../002-librados2_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:11.878 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librados2 (17.2.5-727-g49f8fb05-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-24T18:27:12.582 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs2. 2023-01-24T18:27:12.599 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../003-libcephfs2_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:12.640 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:13.009 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rados. 2023-01-24T18:27:13.024 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../004-python3-rados_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:13.067 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rados (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:13.453 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-01-24T18:27:13.472 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:13.511 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-argparse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:13.772 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cephfs. 2023-01-24T18:27:13.788 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../006-python3-cephfs_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:13.830 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cephfs (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:14.191 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-common. 2023-01-24T18:27:14.208 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:14.249 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:14.536 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-prettytable. 2023-01-24T18:27:14.554 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-01-24T18:27:14.602 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-01-24T18:27:14.855 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rbd. 2023-01-24T18:27:14.872 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../009-python3-rbd_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:14.913 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rbd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:15.282 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-01-24T18:27:15.298 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-01-24T18:27:15.340 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-24T18:27:15.710 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-01-24T18:27:15.727 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-01-24T18:27:15.768 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-01-24T18:27:16.130 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-01-24T18:27:16.146 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-01-24T18:27:16.188 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-24T18:27:16.583 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-01-24T18:27:16.599 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-01-24T18:27:16.644 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-24T18:27:17.010 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-01-24T18:27:17.027 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-01-24T18:27:17.068 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-24T18:27:17.472 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-01-24T18:27:17.488 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-24T18:27:17.532 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T18:27:18.283 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-01-24T18:27:18.300 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-24T18:27:18.341 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T18:27:18.717 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-01-24T18:27:18.734 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-24T18:27:18.775 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T18:27:19.186 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-01-24T18:27:19.206 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-01-24T18:27:19.244 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-24T18:27:19.588 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librgw2. 2023-01-24T18:27:19.606 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../019-librgw2_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:19.647 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librgw2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:20.391 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rgw. 2023-01-24T18:27:20.410 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../020-python3-rgw_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:20.449 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:20.818 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liboath0. 2023-01-24T18:27:20.835 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-01-24T18:27:20.876 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-01-24T18:27:21.172 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libradosstriper1. 2023-01-24T18:27:21.188 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../022-libradosstriper1_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:21.222 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libradosstriper1 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:21.700 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-common. 2023-01-24T18:27:21.716 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../023-ceph-common_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:21.758 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-base. 2023-01-24T18:27:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../024-ceph-base_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-base (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:25.367 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cryptography. 2023-01-24T18:27:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-01-24T18:27:25.416 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-24T18:27:25.796 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-openssl. 2023-01-24T18:27:25.811 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-01-24T18:27:25.845 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-01-24T18:27:26.123 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-01-24T18:27:26.139 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:26.180 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-modules-core (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:26.585 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-bcrypt. 2023-01-24T18:27:26.604 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-01-24T18:27:26.643 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-24T18:27:26.971 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cherrypy3. 2023-01-24T18:27:26.988 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-01-24T18:27:27.029 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-01-24T18:27:27.400 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-natsort. 2023-01-24T18:27:27.416 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-01-24T18:27:27.458 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-01-24T18:27:27.786 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jwt. 2023-01-24T18:27:27.803 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-01-24T18:27:27.844 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-24T18:27:28.130 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-logutils. 2023-01-24T18:27:28.149 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-01-24T18:27:28.205 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-01-24T18:27:28.508 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-markupsafe. 2023-01-24T18:27:28.524 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-01-24T18:27:28.566 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-01-24T18:27:28.894 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-mako. 2023-01-24T18:27:28.911 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-01-24T18:27:28.952 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-24T18:27:29.299 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplegeneric. 2023-01-24T18:27:29.316 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-01-24T18:27:29.358 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-01-24T18:27:29.685 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-singledispatch. 2023-01-24T18:27:29.701 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-01-24T18:27:29.744 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-01-24T18:27:30.063 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webob. 2023-01-24T18:27:30.082 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-01-24T18:27:30.138 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-01-24T18:27:30.457 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-soupsieve. 2023-01-24T18:27:30.473 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-01-24T18:27:30.532 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-24T18:27:30.793 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-bs4. 2023-01-24T18:27:30.812 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-01-24T18:27:30.843 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-01-24T18:27:31.172 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-waitress. 2023-01-24T18:27:31.190 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-01-24T18:27:31.233 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-24T18:27:31.566 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-tempita. 2023-01-24T18:27:31.586 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-01-24T18:27:31.624 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-01-24T18:27:31.910 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-paste. 2023-01-24T18:27:31.930 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-01-24T18:27:31.968 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-01-24T18:27:32.280 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-01-24T18:27:32.296 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-01-24T18:27:32.358 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-01-24T18:27:32.649 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastedeploy. 2023-01-24T18:27:32.670 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-01-24T18:27:32.707 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-01-24T18:27:33.018 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webtest. 2023-01-24T18:27:33.038 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-01-24T18:27:33.076 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-01-24T18:27:33.405 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pecan. 2023-01-24T18:27:33.422 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-01-24T18:27:33.463 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-01-24T18:27:34.542 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-werkzeug. 2023-01-24T18:27:34.562 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2023-01-24T18:27:34.600 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-24T18:27:34.970 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-01-24T18:27:34.987 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:35.028 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libsqlite3-mod-ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:35.406 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr. 2023-01-24T18:27:35.423 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../049-ceph-mgr_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:35.464 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:35.959 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mon. 2023-01-24T18:27:35.976 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../050-ceph-mon_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:36.040 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mon (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:37.171 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-osd. 2023-01-24T18:27:37.191 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../051-ceph-osd_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:37.229 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-osd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:39.990 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph. 2023-01-24T18:27:40.007 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../052-ceph_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:40.048 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:40.351 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-fuse. 2023-01-24T18:27:40.370 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../053-ceph-fuse_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:40.400 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:40.862 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mds. 2023-01-24T18:27:40.879 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../054-ceph-mds_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:40.920 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mds (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:41.490 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package cephadm. 2023-01-24T18:27:41.507 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../055-cephadm_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:41.548 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:41.884 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jinja2. 2023-01-24T18:27:41.902 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-01-24T18:27:41.934 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-01-24T18:27:42.270 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-01-24T18:27:42.290 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:42.328 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:42.690 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-repoze.lru. 2023-01-24T18:27:42.707 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-01-24T18:27:42.747 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-01-24T18:27:43.011 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjs-underscore. 2023-01-24T18:27:43.030 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-01-24T18:27:43.075 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-24T18:27:43.352 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-01-24T18:27:43.368 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-01-24T18:27:43.411 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-24T18:27:43.738 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-routes. 2023-01-24T18:27:43.755 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-01-24T18:27:43.796 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-01-24T18:27:44.149 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-01-24T18:27:44.165 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:44.207 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-dashboard (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:44.703 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-decorator. 2023-01-24T18:27:44.719 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-01-24T18:27:44.761 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-01-24T18:27:45.097 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-01-24T18:27:45.113 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-01-24T18:27:45.155 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-24T18:27:45.516 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-scipy. 2023-01-24T18:27:45.532 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-01-24T18:27:45.574 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-01-24T18:27:46.882 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-01-24T18:27:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-01-24T18:27:46.940 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-24T18:27:47.502 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-joblib. 2023-01-24T18:27:47.519 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-01-24T18:27:47.560 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-01-24T18:27:47.947 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn. 2023-01-24T18:27:47.964 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-01-24T18:27:48.005 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-24T18:27:48.784 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-01-24T18:27:48.801 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:48.876 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:50.179 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cachetools. 2023-01-24T18:27:50.196 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-01-24T18:27:50.237 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-01-24T18:27:50.565 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1. 2023-01-24T18:27:50.583 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-01-24T18:27:50.623 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-01-24T18:27:50.985 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-01-24T18:27:51.003 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-01-24T18:27:51.043 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-24T18:27:51.364 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rsa. 2023-01-24T18:27:51.383 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-01-24T18:27:51.422 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rsa (4.0-3) ... 2023-01-24T18:27:51.758 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-google-auth. 2023-01-24T18:27:51.775 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-01-24T18:27:51.816 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-01-24T18:27:52.164 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-blinker. 2023-01-24T18:27:52.182 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-01-24T18:27:52.222 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-24T18:27:52.526 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-oauthlib. 2023-01-24T18:27:52.546 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-01-24T18:27:52.584 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-24T18:27:52.930 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-01-24T18:27:52.948 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-01-24T18:27:52.988 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-24T18:27:53.333 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-websocket. 2023-01-24T18:27:53.349 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-01-24T18:27:53.391 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-01-24T18:27:53.752 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-kubernetes. 2023-01-24T18:27:53.773 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-01-24T18:27:53.810 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-01-24T18:27:54.365 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-01-24T18:27:54.383 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:27:54.423 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-k8sevents (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:27:54.853 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libonig5:amd64. 2023-01-24T18:27:54.874 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-01-24T18:27:54.911 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-01-24T18:27:55.281 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjq1:amd64. 2023-01-24T18:27:55.298 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-24T18:27:55.340 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-24T18:27:55.626 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package jq. 2023-01-24T18:27:55.642 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-24T18:27:55.684 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-01-24T18:27:55.945 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package socat. 2023-01-24T18:27:55.961 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-01-24T18:27:56.003 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking socat (1.7.3.3-2) ... 2023-01-24T18:27:56.324 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package xmlstarlet. 2023-01-24T18:27:56.343 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-01-24T18:27:56.398 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-01-24T18:27:56.728 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-test. 2023-01-24T18:27:56.745 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../086-ceph-test_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:27:56.803 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-test (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:01.838 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-volume. 2023-01-24T18:28:01.855 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../087-ceph-volume_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T18:28:01.896 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-volume (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:02.264 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package fonts-mathjax. 2023-01-24T18:28:02.281 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-24T18:28:02.323 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-24T18:28:02.848 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs-dev. 2023-01-24T18:28:02.867 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:28:02.906 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs-dev (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:03.325 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package nvme-cli. 2023-01-24T18:28:03.342 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-01-24T18:28:03.401 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-01-24T18:28:03.789 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjs-mathjax. 2023-01-24T18:28:03.810 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-24T18:28:03.847 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-24T18:28:05.528 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-natsort-doc. 2023-01-24T18:28:05.548 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-01-24T18:28:05.586 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-01-24T18:28:05.956 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-atomicwrites. 2023-01-24T18:28:05.973 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-01-24T18:28:06.014 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-01-24T18:28:06.321 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-attr. 2023-01-24T18:28:06.333 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-01-24T18:28:06.375 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-01-24T18:28:06.736 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webencodings. 2023-01-24T18:28:06.753 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-01-24T18:28:06.794 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-24T18:28:07.130 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-html5lib. 2023-01-24T18:28:07.148 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-01-24T18:28:07.197 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-01-24T18:28:07.558 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-01-24T18:28:07.576 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-01-24T18:28:07.616 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-24T18:28:08.088 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-packaging. 2023-01-24T18:28:08.107 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-01-24T18:28:08.145 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-packaging (20.3-1) ... 2023-01-24T18:28:08.464 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastescript. 2023-01-24T18:28:08.481 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-01-24T18:28:08.522 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-01-24T18:28:08.876 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pluggy. 2023-01-24T18:28:08.892 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-01-24T18:28:08.933 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-01-24T18:28:09.278 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-psutil. 2023-01-24T18:28:09.296 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-01-24T18:28:09.336 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-01-24T18:28:09.681 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-py. 2023-01-24T18:28:09.700 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-01-24T18:28:09.739 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-01-24T18:28:10.093 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyinotify. 2023-01-24T18:28:10.109 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-01-24T18:28:10.151 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-24T18:28:10.495 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-wcwidth. 2023-01-24T18:28:10.513 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-01-24T18:28:10.553 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-24T18:28:10.865 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pytest. 2023-01-24T18:28:10.882 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-01-24T18:28:10.956 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-01-24T18:28:11.293 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplejson. 2023-01-24T18:28:11.310 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-01-24T18:28:11.351 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-24T18:28:11.678 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-01-24T18:28:11.695 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-01-24T18:28:11.736 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-24T18:28:12.499 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package radosgw. 2023-01-24T18:28:12.519 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../108-radosgw_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:28:12.557 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking radosgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:14.244 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package rbd-fuse. 2023-01-24T18:28:14.261 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../109-rbd-fuse_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T18:28:14.302 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking rbd-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:14.632 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-attr (19.3.0-2) ... 2023-01-24T18:28:14.932 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-01-24T18:28:15.091 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-24T18:28:15.192 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-01-24T18:28:15.369 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:16.038 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user cephadm....done 2023-01-24T18:28:16.192 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-24T18:28:16.293 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-24T18:28:16.670 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-01-24T18:28:16.979 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-24T18:28:17.105 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-01-24T18:28:17.265 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-01-24T18:28:17.591 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-01-24T18:28:17.992 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-24T18:28:18.285 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-01-24T18:28:18.598 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-24T18:28:18.887 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-24T18:28:19.180 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-argparse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:19.515 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-24T18:28:19.641 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-01-24T18:28:19.958 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-24T18:28:20.559 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-24T18:28:20.685 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-01-24T18:28:21.062 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-01-24T18:28:21.371 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-01-24T18:28:21.681 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-24T18:28:22.032 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-01-24T18:28:22.316 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-01-24T18:28:22.667 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-01-24T18:28:22.960 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-packaging (20.3-1) ... 2023-01-24T18:28:23.244 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-01-24T18:28:23.404 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-24T18:28:23.538 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-24T18:28:23.831 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-01-24T18:28:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-24T18:28:24.646 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-24T18:28:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up socat (1.7.3.3-2) ... 2023-01-24T18:28:25.170 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:25.482 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T18:28:25.641 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-01-24T18:28:26.009 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-24T18:28:26.135 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-01-24T18:28:26.261 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-01-24T18:28:26.554 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-24T18:28:26.855 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T18:28:27.006 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-24T18:28:27.132 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-01-24T18:28:27.458 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-01-24T18:28:27.743 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-01-24T18:28:28.052 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-24T18:28:28.170 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-01-24T18:28:28.479 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-01-24T18:28:28.764 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-24T18:28:29.056 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-01-24T18:28:29.458 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-01-24T18:28:29.692 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-01-24T18:28:29.818 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-24T18:28:30.128 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-01-24T18:28:30.479 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-24T18:28:30.639 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-24T18:28:30.765 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-01-24T18:28:31.107 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-24T18:28:31.400 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librados2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:31.551 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-01-24T18:28:33.023 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libsqlite3-mod-ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:33.149 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-24T18:28:33.300 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-01-24T18:28:33.710 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-01-24T18:28:34.011 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-01-24T18:28:34.296 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T18:28:34.430 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-01-24T18:28:34.773 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:34.899 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-24T18:28:35.226 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libradosstriper1 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:35.352 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-24T18:28:35.678 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-24T18:28:36.742 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librbd1 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:36.884 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-24T18:28:37.010 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-01-24T18:28:37.136 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-01-24T18:28:37.279 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rsa (4.0-3) ... 2023-01-24T18:28:37.588 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-24T18:28:37.732 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-01-24T18:28:38.040 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-modules-core (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:38.206 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:38.427 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-24T18:28:38.427 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-24T18:28:38.883 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs-dev (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:39.119 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-24T18:28:39.429 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-01-24T18:28:39.739 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-01-24T18:28:40.100 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rados (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:40.226 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librgw2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:40.352 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rbd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:40.503 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up rbd-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:40.621 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-routes (2.4.1-2) ... 2023-01-24T18:28:40.964 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-01-24T18:28:41.257 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:41.425 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cephfs (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:41.551 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-01-24T18:28:42.464 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:43.027 INFO:teuthology.orchestra.run.smithi082.stdout:Adding group ceph....done 2023-01-24T18:28:43.437 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user ceph....done 2023-01-24T18:28:43.613 INFO:teuthology.orchestra.run.smithi082.stdout:Setting system user ceph properties....done 2023-01-24T18:28:43.633 INFO:teuthology.orchestra.run.smithi082.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-01-24T18:28:43.773 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-01-24T18:28:44.084 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-01-24T18:28:44.534 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-test (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:44.685 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up radosgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:44.959 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-24T18:28:44.959 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-24T18:28:45.712 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-base (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:46.524 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-01-24T18:28:46.982 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mds (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:47.211 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-24T18:28:47.212 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-24T18:28:47.651 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:47.858 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-24T18:28:47.858 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-24T18:28:48.286 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-osd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:48.796 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-24T18:28:48.797 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-24T18:28:49.229 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-k8sevents (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:49.355 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-diskprediction-local (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:49.498 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mon (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:49.724 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-24T18:28:49.725 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-24T18:28:50.166 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:50.309 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:50.434 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-dashboard (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:50.602 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-volume (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T18:28:50.729 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-24T18:28:50.881 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-01-24T18:28:51.174 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-24T18:28:54.010 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-01-24T18:28:54.574 INFO:teuthology.orchestra.run.smithi082.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-24T18:28:54.577 DEBUG:teuthology.parallel:result is None 2023-01-24T18:28:54.577 DEBUG:teuthology.orchestra.run.smithi082:> sudo lsb_release -is 2023-01-24T18:28:54.626 INFO:teuthology.orchestra.run.smithi082.stdout:Ubuntu 2023-01-24T18:28:54.631 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-24T18:28:54.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T18:28:54.790 DEBUG:teuthology.orchestra.run.smithi082:> dpkg-query -W -f '${Version}' ceph 2023-01-24T18:28:54.814 INFO:teuthology.orchestra.run.smithi082.stdout:17.2.5-727-g49f8fb05-1focal 2023-01-24T18:28:54.815 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-727-g49f8fb05-1focal 2023-01-24T18:28:54.816 INFO:teuthology.task.install:The correct ceph version 17.2.5-727-g49f8fb05-1focal is installed. 2023-01-24T18:28:54.817 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T18:28:54.817 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:28:54.818 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T18:28:54.828 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T18:28:54.829 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:28:54.829 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T18:28:54.888 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T18:28:54.904 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T18:28:54.904 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:28:54.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T18:28:54.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T18:28:54.972 INFO:teuthology.run_tasks:Running task workunit... 2023-01-24T18:28:54.985 INFO:tasks.workunit:Pulling workunits from ref 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T18:28:54.986 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-24T18:28:54.986 INFO:tasks.workunit:timeout=3h 2023-01-24T18:28:54.986 INFO:tasks.workunit:cleanup=True 2023-01-24T18:28:54.987 DEBUG:teuthology.orchestra.run.smithi082:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-24T18:28:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T18:28:54.994 INFO:teuthology.orchestra.run.smithi082.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-24T18:28:54.995 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-24T18:28:55.043 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-24T18:28:55.043 DEBUG:teuthology.orchestra.run.smithi082:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-24T18:28:55.050 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T18:28:55.058 INFO:tasks.workunit.client.0.smithi082.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-24T18:29:37.425 INFO:tasks.workunit.client.0.smithi082.stderr:Note: switching to '49f8fb05584886826e8eade75f7105fba754560c'. 2023-01-24T18:29:37.426 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.426 INFO:tasks.workunit.client.0.smithi082.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-24T18:29:37.426 INFO:tasks.workunit.client.0.smithi082.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-24T18:29:37.426 INFO:tasks.workunit.client.0.smithi082.stderr:state without impacting any branches by switching back to a branch. 2023-01-24T18:29:37.426 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.427 INFO:tasks.workunit.client.0.smithi082.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-24T18:29:37.427 INFO:tasks.workunit.client.0.smithi082.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-24T18:29:37.427 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.427 INFO:tasks.workunit.client.0.smithi082.stderr: git switch -c 2023-01-24T18:29:37.427 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.428 INFO:tasks.workunit.client.0.smithi082.stderr:Or undo this operation with: 2023-01-24T18:29:37.428 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.428 INFO:tasks.workunit.client.0.smithi082.stderr: git switch - 2023-01-24T18:29:37.428 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.428 INFO:tasks.workunit.client.0.smithi082.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-24T18:29:37.429 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:29:37.429 INFO:tasks.workunit.client.0.smithi082.stderr:HEAD is now at 49f8fb05584 Merge branch 'wip-58543-quincy' of https://github.com/ivancich/ceph-fork into wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T18:29:37.430 DEBUG:teuthology.orchestra.run.smithi082:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-24T18:29:37.440 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-01-24T18:29:37.440 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-24T18:29:37.446 INFO:tasks.workunit:Running workunits matching osd-backfill on client.0... 2023-01-24T18:29:37.447 INFO:tasks.workunit:Running workunit osd-backfill/osd-backfill-prio.sh... 2023-01-24T18:29:37.447 DEBUG:teuthology.orchestra.run.smithi082:workunit test osd-backfill/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49f8fb05584886826e8eade75f7105fba754560c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh 2023-01-24T18:29:37.523 INFO:tasks.workunit.client.0.smithi082.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-01-24T18:29:37.536 INFO:tasks.workunit.client.0.smithi082.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-01-24T18:29:37.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-24T18:29:37.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-24T18:29:37.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: export PYTHONWARNINGS=ignore 2023-01-24T18:29:37.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: PYTHONWARNINGS=ignore 2023-01-24T18:29:37.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export CEPH_CONF=/dev/null 2023-01-24T18:29:37.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: CEPH_CONF=/dev/null 2023-01-24T18:29:37.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: unset CEPH_ARGS 2023-01-24T18:29:37.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: local code 2023-01-24T18:29:37.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: main: run td/osd-backfill-prio 2023-01-24T18:29:37.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2023-01-24T18:29:37.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:22: run: shift 2023-01-24T18:29:37.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2023-01-24T18:29:37.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2023-01-24T18:29:37.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2023-01-24T18:29:37.540 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:27: run: uuidgen 2023-01-24T18:29:37.540 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none ' 2023-01-24T18:29:37.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2023-01-24T18:29:37.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2023-01-24T18:29:37.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:32: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2023-01-24T18:29:37.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:33: run: export objects=50 2023-01-24T18:29:37.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:33: run: objects=50 2023-01-24T18:29:37.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:34: run: export poolprefix=test 2023-01-24T18:29:37.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:34: run: poolprefix=test 2023-01-24T18:29:37.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:35: run: export FORCE_PRIO=254 2023-01-24T18:29:37.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:35: run: FORCE_PRIO=254 2023-01-24T18:29:37.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:36: run: export DEGRADED_PRIO=150 2023-01-24T18:29:37.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:36: run: DEGRADED_PRIO=150 2023-01-24T18:29:37.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:37: run: export NORMAL_PRIO=110 2023-01-24T18:29:37.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:37: run: NORMAL_PRIO=110 2023-01-24T18:29:37.543 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: set 2023-01-24T18:29:37.543 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-01-24T18:29:37.544 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: local 'funcs=TEST_backfill_pool_priority 2023-01-24T18:29:37.545 INFO:tasks.workunit.client.0.smithi082.stderr:TEST_backfill_priority' 2023-01-24T18:29:37.545 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:40: run: for func in $funcs 2023-01-24T18:29:37.545 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:41: run: setup td/osd-backfill-prio 2023-01-24T18:29:37.545 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2023-01-24T18:29:37.545 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2023-01-24T18:29:37.545 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2023-01-24T18:29:37.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-24T18:29:37.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2023-01-24T18:29:37.546 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-24T18:29:37.546 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-24T18:29:37.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-24T18:29:37.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-24T18:29:37.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-24T18:29:37.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-24T18:29:37.548 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-24T18:29:37.549 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-24T18:29:37.549 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-24T18:29:37.551 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-24T18:29:37.552 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-24T18:29:37.552 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-24T18:29:37.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:29:37.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-24T18:29:37.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-24T18:29:37.554 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:29:37.554 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-24T18:29:37.555 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-24T18:29:37.555 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2023-01-24T18:29:37.557 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-24T18:29:37.557 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:37.558 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:37.558 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.23029 2023-01-24T18:29:37.558 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-24T18:29:37.559 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-24T18:29:37.559 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2023-01-24T18:29:37.560 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-24T18:29:37.561 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:37.561 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:37.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.23029 2023-01-24T18:29:37.563 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-24T18:29:37.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-01-24T18:29:37.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-01-24T18:29:37.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-24T18:29:37.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2023-01-24T18:29:37.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: TEST_backfill_pool_priority td/osd-backfill-prio 2023-01-24T18:29:37.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:329: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2023-01-24T18:29:37.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: local pools=3 2023-01-24T18:29:37.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: local OSDS=2 2023-01-24T18:29:37.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:333: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2023-01-24T18:29:37.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2023-01-24T18:29:37.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-24T18:29:37.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-24T18:29:37.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-24T18:29:37.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2023-01-24T18:29:37.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2023-01-24T18:29:38.058 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-24T18:29:38.059 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:29:38.059 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:29:38.060 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:29:38.060 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:38.060 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:38.060 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:29:38.061 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-01-24T18:29:38.357 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-24T18:29:38.360 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-24T18:29:38.360 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-24T18:29:38.361 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-24T18:29:38.361 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-24T18:29:38.361 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-24T18:29:38.361 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-24T18:29:38.361 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-24T18:29:38.361 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-24T18:29:38.362 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:29:38.362 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:38.362 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:38.363 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-mon.a.asok 2023-01-24T18:29:38.363 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-24T18:29:38.363 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.23029/ceph-mon.a.asok config get fsid 2023-01-24T18:29:38.449 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-24T18:29:38.450 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-24T18:29:38.450 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-24T18:29:38.450 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-24T18:29:38.451 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-24T18:29:38.452 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-24T18:29:38.452 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-24T18:29:38.452 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-24T18:29:38.452 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:29:38.453 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:38.453 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:38.453 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-mon.a.asok 2023-01-24T18:29:38.453 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-24T18:29:38.453 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.23029/ceph-mon.a.asok config get mon_host 2023-01-24T18:29:38.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2023-01-24T18:29:38.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-prio 2023-01-24T18:29:38.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-24T18:29:38.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-24T18:29:38.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-24T18:29:38.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-prio/x 2023-01-24T18:29:38.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-24T18:29:39.440 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-24T18:29:39.441 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:29:39.441 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:29:39.442 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:29:39.442 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:39.442 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:39.442 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:29:39.443 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-24T18:29:39.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-24T18:29:39.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:335: TEST_backfill_pool_priority: export CEPH_ARGS 2023-01-24T18:29:39.603 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: expr 2 - 1 2023-01-24T18:29:39.604 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: seq 0 1 2023-01-24T18:29:39.608 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:29:39.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2023-01-24T18:29:39.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:29:39.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:29:39.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-24T18:29:39.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:29:39.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/0 2023-01-24T18:29:39.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:29:39.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:29:39.611 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:29:39.611 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:29:39.611 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2023-01-24T18:29:39.611 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2023-01-24T18:29:39.612 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:29:39.612 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:29:39.612 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:29:39.612 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:29:39.612 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:29:39.613 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:29:39.614 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:29:39.615 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:39.615 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:39.615 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:29:39.615 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:29:39.615 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:29:39.616 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:29:39.616 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:29:39.616 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:29:39.616 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:29:39.616 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:29:39.617 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:29:39.617 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:29:39.617 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:29:39.617 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:29:39.617 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/0 2023-01-24T18:29:39.618 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:29:39.618 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=146c53bf-4b15-4bb9-b1d0-65215695f471 2023-01-24T18:29:39.618 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 146c53bf-4b15-4bb9-b1d0-65215695f471' 2023-01-24T18:29:39.618 INFO:tasks.workunit.client.0.smithi082.stdout:add osd0 146c53bf-4b15-4bb9-b1d0-65215695f471 2023-01-24T18:29:39.619 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:29:39.639 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCTI9Bj1y6wJRAAAXY300F4YFm9sBAana8umA== 2023-01-24T18:29:39.639 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCTI9Bj1y6wJRAAAXY300F4YFm9sBAana8umA=="}' 2023-01-24T18:29:39.639 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 146c53bf-4b15-4bb9-b1d0-65215695f471 -i td/osd-backfill-prio/0/new.json 2023-01-24T18:29:40.005 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:29:40.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/0/new.json 2023-01-24T18:29:40.018 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCTI9Bj1y6wJRAAAXY300F4YFm9sBAana8umA== --osd-uuid 146c53bf-4b15-4bb9-b1d0-65215695f471 2023-01-24T18:29:40.103 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.096+0000 7f04867dc3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:40.129 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.124+0000 7f04867dc3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:40.131 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.124+0000 7f04867dc3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:40.132 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.124+0000 7f04867dc3c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2023-01-24T18:29:40.132 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.124+0000 7f04867dc3c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2023-01-24T18:29:40.133 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.124+0000 7f04867dc3c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2023-01-24T18:29:40.229 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:40.220+0000 7f04867dc3c0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2023-01-24T18:29:46.535 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2023-01-24T18:29:46.535 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:29:46.536 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-24T18:29:46.537 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd0 key to auth repository 2023-01-24T18:29:46.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:29:46.766 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-24T18:29:46.766 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.0 2023-01-24T18:29:46.767 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:29:46.767 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:29:46.767 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:29:46.768 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:29:46.795 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:46.788+0000 7fdaf2e7d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:46.815 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:46.808+0000 7fdaf2e7d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:46.816 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:46.808+0000 7fdaf2e7d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:47.031 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-24T18:29:47.031 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:29:47.032 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-24T18:29:47.032 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:29:47.032 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:29:47.032 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:29:47.032 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:29:47.033 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:29:47.033 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:29:47.033 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:29:47.269 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:29:47.391 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:47.384+0000 7fdaf2e7d3c0 -1 Falling back to public interface 2023-01-24T18:29:48.271 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:29:48.271 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:29:48.271 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:29:48.272 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:29:48.272 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:29:48.272 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:29:48.714 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:29:49.716 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:29:49.716 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:29:49.717 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:29:49.717 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:29:49.717 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:29:49.718 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:29:50.106 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:50.100+0000 7fdaf2e7d3c0 -1 osd.0 0 log_to_monitors true 2023-01-24T18:29:50.182 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:29:51.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:29:51.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:29:51.185 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:29:51.186 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:29:51.186 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:29:51.186 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:29:51.644 INFO:tasks.workunit.client.0.smithi082.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/24075,v1:127.0.0.1:6801/24075] [v2:127.0.0.1:6802/24075,v1:127.0.0.1:6803/24075] exists,up 146c53bf-4b15-4bb9-b1d0-65215695f471 2023-01-24T18:29:51.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:29:51.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:29:51.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:29:51.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:29:51.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2023-01-24T18:29:51.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:29:51.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:29:51.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-24T18:29:51.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:29:51.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/1 2023-01-24T18:29:51.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:29:51.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:29:51.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:29:51.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:29:51.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2023-01-24T18:29:51.651 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2023-01-24T18:29:51.651 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:29:51.651 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:29:51.651 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:29:51.651 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:29:51.652 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:29:51.652 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:29:51.652 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:29:51.652 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:29:51.652 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:29:51.653 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:29:51.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:29:51.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:29:51.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:29:51.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:29:51.654 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:29:51.654 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:29:51.654 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:29:51.654 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:29:51.654 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:29:51.655 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:29:51.655 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:29:51.655 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/1 2023-01-24T18:29:51.656 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:29:51.656 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=51295745-9f82-44b6-b1f0-086845f521ed 2023-01-24T18:29:51.656 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 51295745-9f82-44b6-b1f0-086845f521ed' 2023-01-24T18:29:51.656 INFO:tasks.workunit.client.0.smithi082.stdout:add osd1 51295745-9f82-44b6-b1f0-086845f521ed 2023-01-24T18:29:51.657 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:29:51.676 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCfI9BjuZr0JxAAYFoZ3mUeK9dnlMRr9rGyqw== 2023-01-24T18:29:51.676 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCfI9BjuZr0JxAAYFoZ3mUeK9dnlMRr9rGyqw=="}' 2023-01-24T18:29:51.676 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 51295745-9f82-44b6-b1f0-086845f521ed -i td/osd-backfill-prio/1/new.json 2023-01-24T18:29:52.195 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:29:52.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/1/new.json 2023-01-24T18:29:52.207 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCfI9BjuZr0JxAAYFoZ3mUeK9dnlMRr9rGyqw== --osd-uuid 51295745-9f82-44b6-b1f0-086845f521ed 2023-01-24T18:29:52.247 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.240+0000 7f5cc09ec3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:52.299 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.292+0000 7f5cc09ec3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:52.301 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.296+0000 7f5cc09ec3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:52.302 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.296+0000 7f5cc09ec3c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2023-01-24T18:29:52.303 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.296+0000 7f5cc09ec3c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2023-01-24T18:29:52.303 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.296+0000 7f5cc09ec3c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2023-01-24T18:29:52.391 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:52.384+0000 7f5cc09ec3c0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2023-01-24T18:29:58.416 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2023-01-24T18:29:58.416 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:29:58.418 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-24T18:29:58.418 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd1 key to auth repository 2023-01-24T18:29:58.419 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:29:58.993 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-24T18:29:58.994 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.1 2023-01-24T18:29:58.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:29:58.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:29:58.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:29:58.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:29:59.022 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:59.016+0000 7f0e6c92b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:59.024 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:59.016+0000 7f0e6c92b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:59.025 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:59.020+0000 7f0e6c92b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:29:59.483 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-24T18:29:59.483 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:29:59.483 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-24T18:29:59.483 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:29:59.484 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:29:59.484 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:29:59.484 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:29:59.484 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:29:59.485 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:29:59.485 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:29:59.571 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:29:59.563+0000 7f0e6c92b3c0 -1 Falling back to public interface 2023-01-24T18:29:59.985 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:30:00.986 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:30:00.987 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:30:00.987 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:30:00.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:30:00.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:30:00.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:30:01.392 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:30:02.083 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:30:02.075+0000 7f0e6c92b3c0 -1 osd.1 0 log_to_monitors true 2023-01-24T18:30:02.394 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:30:02.394 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:30:02.394 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:30:02.395 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:30:02.395 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:30:02.396 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:30:02.890 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:30:03.892 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:30:03.893 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:30:03.893 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:30:03.894 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:30:03.894 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:30:03.894 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:30:04.307 INFO:tasks.workunit.client.0.smithi082.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25547,v1:127.0.0.1:6811/25547] [v2:127.0.0.1:6812/25547,v1:127.0.0.1:6813/25547] exists,up 51295745-9f82-44b6-b1f0-086845f521ed 2023-01-24T18:30:04.307 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:30:04.308 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:30:04.308 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:30:04.309 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: seq 1 3 2023-01-24T18:30:04.310 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:04.310 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test1 1 1 2023-01-24T18:30:04.310 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2023-01-24T18:30:04.969 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test1' already exists 2023-01-24T18:30:04.977 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:30:05.979 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2023-01-24T18:30:07.697 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 1 size to 2 2023-01-24T18:30:07.711 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:07.712 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test2 1 1 2023-01-24T18:30:07.712 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2023-01-24T18:30:09.381 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test2' already exists 2023-01-24T18:30:09.389 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:30:10.391 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2023-01-24T18:30:11.683 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 2 size to 2 2023-01-24T18:30:11.699 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:11.699 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test3 1 1 2023-01-24T18:30:11.700 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2023-01-24T18:30:12.405 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test3' already exists 2023-01-24T18:30:12.417 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:30:13.419 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2023-01-24T18:30:14.455 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 3 size to 2 2023-01-24T18:30:14.468 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:347: TEST_backfill_pool_priority: sleep 5 2023-01-24T18:30:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:349: TEST_backfill_pool_priority: wait_for_clean 2023-01-24T18:30:19.470 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-24T18:30:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-24T18:30:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-24T18:30:19.471 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-24T18:30:19.471 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-24T18:30:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-24T18:30:19.472 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-24T18:30:19.473 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-24T18:30:19.473 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-24T18:30:19.473 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-24T18:30:19.695 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-24T18:30:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-24T18:30:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-24T18:30:19.696 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:30:19.697 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:30:20.170 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:30:20.170 INFO:tasks.workunit.client.0.smithi082.stderr:1' 2023-01-24T18:30:20.170 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:30:20.170 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:30:20.171 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:30:20.372 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836488 2023-01-24T18:30:20.372 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836488 2023-01-24T18:30:20.372 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836488' 2023-01-24T18:30:20.372 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:30:20.373 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:30:20.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672966 2023-01-24T18:30:20.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672966 2023-01-24T18:30:20.533 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2023-01-24T18:30:20.533 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:30:20.533 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836488 2023-01-24T18:30:20.534 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:30:20.536 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:30:20.537 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836488 2023-01-24T18:30:20.537 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:30:20.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836488 2023-01-24T18:30:20.539 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836488 2023-01-24T18:30:20.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2023-01-24T18:30:20.540 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:30:21.011 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836486 -lt 21474836488 2023-01-24T18:30:21.012 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:30:22.013 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:30:22.014 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:30:22.423 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836489 -lt 21474836488 2023-01-24T18:30:22.424 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:30:22.425 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:30:22.425 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672966 2023-01-24T18:30:22.426 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:30:22.428 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672966 2023-01-24T18:30:22.428 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:30:22.430 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672966 2023-01-24T18:30:22.430 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949672966 2023-01-24T18:30:22.431 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2023-01-24T18:30:22.431 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:30:22.864 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672966 -lt 42949672966 2023-01-24T18:30:22.865 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-24T18:30:22.865 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:30:22.866 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:30:23.388 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 3 == 0 2023-01-24T18:30:23.389 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:30:23.389 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:30:23.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:30:23.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:30:23.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:30:23.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:30:23.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:30:23.807 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=3 2023-01-24T18:30:23.808 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:30:23.809 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:30:23.809 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:30:24.346 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 3 = 3 2023-01-24T18:30:24.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-24T18:30:24.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-24T18:30:24.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:351: TEST_backfill_pool_priority: ceph pg dump pgs 2023-01-24T18:30:24.719 INFO:tasks.workunit.client.0.smithi082.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-24T18:30:24.719 INFO:tasks.workunit.client.0.smithi082.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:30:14.903042+0000 0'0 26:18 [1,0] 1 [1,0] 1 0'0 2023-01-24T18:30:12.118137+0000 0'0 2023-01-24T18:30:12.118137+0000 0 0 periodic scrub scheduled @ 2023-01-25T19:32:55.214847+0000 0 0 2023-01-24T18:30:24.720 INFO:tasks.workunit.client.0.smithi082.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:30:12.054841+0000 0'0 25:24 [0,1] 0 [0,1] 0 0'0 2023-01-24T18:30:08.213541+0000 0'0 2023-01-24T18:30:08.213541+0000 0 0 periodic scrub scheduled @ 2023-01-25T19:36:47.215791+0000 0 0 2023-01-24T18:30:24.720 INFO:tasks.workunit.client.0.smithi082.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:30:08.717637+0000 0'0 25:27 [1,0] 1 [1,0] 1 0'0 2023-01-24T18:30:04.698077+0000 0'0 2023-01-24T18:30:04.698077+0000 0 0 periodic scrub scheduled @ 2023-01-25T19:00:59.753918+0000 0 0 2023-01-24T18:30:24.720 INFO:tasks.workunit.client.0.smithi082.stdout: 2023-01-24T18:30:24.720 INFO:tasks.workunit.client.0.smithi082.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-24T18:30:24.722 INFO:tasks.workunit.client.0.smithi082.stderr:dumped pgs 2023-01-24T18:30:24.730 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local PG1 2023-01-24T18:30:24.730 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local POOLNUM1 2023-01-24T18:30:24.730 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:357: TEST_backfill_pool_priority: local pool1 2023-01-24T18:30:24.730 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local chk_osd1_1 2023-01-24T18:30:24.731 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local chk_osd1_2 2023-01-24T18:30:24.731 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local PG2 2023-01-24T18:30:24.731 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local POOLNUM2 2023-01-24T18:30:24.731 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:363: TEST_backfill_pool_priority: local pool2 2023-01-24T18:30:24.731 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: local chk_osd2_1 2023-01-24T18:30:24.732 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:365: TEST_backfill_pool_priority: local chk_osd2_2 2023-01-24T18:30:24.732 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: seq 1 3 2023-01-24T18:30:24.732 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:24.733 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2023-01-24T18:30:24.733 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: jq '.acting[]' 2023-01-24T18:30:25.124 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:30:25.125 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: local test_osd1=1 2023-01-24T18:30:25.126 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:30:25.156 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: local test_osd2=0 2023-01-24T18:30:25.157 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: '[' -z '' ']' 2023-01-24T18:30:25.157 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: PG1=1.0 2023-01-24T18:30:25.157 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: POOLNUM1=1 2023-01-24T18:30:25.157 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: pool1=test1 2023-01-24T18:30:25.157 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:377: TEST_backfill_pool_priority: chk_osd1_1=1 2023-01-24T18:30:25.158 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: chk_osd1_2=0 2023-01-24T18:30:25.158 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:25.158 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2023-01-24T18:30:25.158 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: jq '.acting[]' 2023-01-24T18:30:25.555 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:30:25.557 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: local test_osd1=0 2023-01-24T18:30:25.558 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:30:25.560 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: local test_osd2=1 2023-01-24T18:30:25.560 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2023-01-24T18:30:25.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2023-01-24T18:30:25.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: PG2=2.0 2023-01-24T18:30:25.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: POOLNUM2=2 2023-01-24T18:30:25.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: pool2=test2 2023-01-24T18:30:25.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:384: TEST_backfill_pool_priority: chk_osd2_1=0 2023-01-24T18:30:25.562 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:385: TEST_backfill_pool_priority: chk_osd2_2=1 2023-01-24T18:30:25.562 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: break 2023-01-24T18:30:25.562 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:389: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2023-01-24T18:30:25.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:391: TEST_backfill_pool_priority: '[' test2 = '' ']' 2023-01-24T18:30:25.565 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: seq 1 3 2023-01-24T18:30:25.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:25.566 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2023-01-24T18:30:25.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:25.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2023-01-24T18:30:25.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2023-01-24T18:30:25.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2023-01-24T18:30:25.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:401: TEST_backfill_pool_priority: delete_pool test3 2023-01-24T18:30:25.568 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2023-01-24T18:30:25.568 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2023-01-24T18:30:26.118 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test3' does not exist 2023-01-24T18:30:26.125 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_extra_prio=1 2023-01-24T18:30:26.125 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_extra_prio=2 2023-01-24T18:30:26.126 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: expr 150 + 1 + 1 2023-01-24T18:30:26.128 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: pool1_prio=152 2023-01-24T18:30:26.128 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: expr 150 + 1 + 2 2023-01-24T18:30:26.129 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: pool2_prio=153 2023-01-24T18:30:26.129 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2023-01-24T18:30:27.233 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 1 size to 1 2023-01-24T18:30:27.246 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2023-01-24T18:30:27.986 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 1 recovery_priority to 1 2023-01-24T18:30:27.997 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:413: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2023-01-24T18:30:28.898 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 2 size to 1 2023-01-24T18:30:28.910 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2023-01-24T18:30:29.970 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 2 recovery_priority to 2 2023-01-24T18:30:29.982 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: wait_for_clean 2023-01-24T18:30:29.982 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-24T18:30:29.982 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-24T18:30:29.983 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-24T18:30:29.983 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-24T18:30:29.984 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-24T18:30:29.985 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-24T18:30:29.985 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-24T18:30:29.985 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-24T18:30:29.985 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-24T18:30:29.986 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-24T18:30:30.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-24T18:30:30.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-24T18:30:30.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-24T18:30:30.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:30:30.185 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:30:30.591 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:30:30.592 INFO:tasks.workunit.client.0.smithi082.stderr:1' 2023-01-24T18:30:30.592 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:30:30.592 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:30:30.593 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:30:30.763 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836492 2023-01-24T18:30:30.763 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836492 2023-01-24T18:30:30.764 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836492' 2023-01-24T18:30:30.764 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:30:30.764 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:30:30.905 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672970 2023-01-24T18:30:30.906 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672970 2023-01-24T18:30:30.906 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2023-01-24T18:30:30.906 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:30:30.907 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836492 2023-01-24T18:30:30.907 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:30:30.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:30:30.909 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836492 2023-01-24T18:30:30.909 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:30:30.911 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836492 2023-01-24T18:30:30.912 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2023-01-24T18:30:30.912 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836492 2023-01-24T18:30:30.913 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:30:31.283 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836490 -lt 21474836492 2023-01-24T18:30:31.283 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:30:32.285 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:30:32.286 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:30:32.712 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836493 -lt 21474836492 2023-01-24T18:30:32.713 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:30:32.713 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672970 2023-01-24T18:30:32.714 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:30:32.714 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:30:32.716 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672970 2023-01-24T18:30:32.716 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:30:32.717 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672970 2023-01-24T18:30:32.718 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949672970 2023-01-24T18:30:32.718 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2023-01-24T18:30:32.721 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:30:33.122 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672970 -lt 42949672970 2023-01-24T18:30:33.123 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-24T18:30:33.124 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:30:33.124 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:30:33.643 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 2 == 0 2023-01-24T18:30:33.643 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:30:33.644 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:30:33.645 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:30:33.645 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:30:33.645 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:30:33.646 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:30:33.646 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:30:34.041 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=2 2023-01-24T18:30:34.042 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:30:34.042 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:30:34.042 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:30:34.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 2 = 2 2023-01-24T18:30:34.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-24T18:30:34.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-24T18:30:34.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:417: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2023-01-24T18:30:34.596 INFO:tasks.workunit.client.0.smithi082.stderr:10+0 records in 2023-01-24T18:30:34.596 INFO:tasks.workunit.client.0.smithi082.stderr:10+0 records out 2023-01-24T18:30:34.596 INFO:tasks.workunit.client.0.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0309147 s, 339 MB/s 2023-01-24T18:30:34.597 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: p=1 2023-01-24T18:30:34.597 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:419: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2023-01-24T18:30:34.597 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: seq 1 50 2023-01-24T18:30:34.598 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:34.598 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2023-01-24T18:30:35.117 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:35.117 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2023-01-24T18:30:35.518 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:35.518 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2023-01-24T18:30:35.935 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:35.936 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2023-01-24T18:30:36.320 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:36.321 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2023-01-24T18:30:36.738 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:36.738 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2023-01-24T18:30:37.148 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:37.149 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2023-01-24T18:30:37.524 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:37.525 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2023-01-24T18:30:37.950 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:37.951 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2023-01-24T18:30:38.293 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:38.294 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2023-01-24T18:30:38.689 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:38.689 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2023-01-24T18:30:39.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:39.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2023-01-24T18:30:39.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:39.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2023-01-24T18:30:40.016 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:40.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2023-01-24T18:30:40.409 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:40.409 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2023-01-24T18:30:40.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:40.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2023-01-24T18:30:41.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:41.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2023-01-24T18:30:41.805 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:41.805 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2023-01-24T18:30:42.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:42.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2023-01-24T18:30:42.607 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:42.608 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2023-01-24T18:30:43.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:43.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2023-01-24T18:30:43.402 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:43.402 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2023-01-24T18:30:43.820 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:43.820 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2023-01-24T18:30:44.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:44.222 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2023-01-24T18:30:44.624 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:44.625 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2023-01-24T18:30:45.068 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:45.068 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2023-01-24T18:30:45.461 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:45.462 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2023-01-24T18:30:45.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:45.888 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2023-01-24T18:30:46.289 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:46.289 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2023-01-24T18:30:46.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:46.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2023-01-24T18:30:47.042 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:47.042 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2023-01-24T18:30:47.435 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:47.435 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2023-01-24T18:30:47.903 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:47.903 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2023-01-24T18:30:48.321 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:48.321 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2023-01-24T18:30:48.737 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:48.737 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2023-01-24T18:30:49.166 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:49.167 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2023-01-24T18:30:49.559 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:49.560 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2023-01-24T18:30:49.994 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:49.994 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2023-01-24T18:30:50.395 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:50.396 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2023-01-24T18:30:50.797 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:50.797 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2023-01-24T18:30:51.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:51.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2023-01-24T18:30:51.575 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:51.575 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2023-01-24T18:30:52.018 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:52.018 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2023-01-24T18:30:52.427 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:52.428 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2023-01-24T18:30:52.904 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:52.905 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2023-01-24T18:30:53.280 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:53.281 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2023-01-24T18:30:53.673 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:53.674 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2023-01-24T18:30:54.104 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:54.104 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2023-01-24T18:30:54.501 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:54.501 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2023-01-24T18:30:54.945 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:54.945 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2023-01-24T18:30:55.314 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: expr 1 + 1 2023-01-24T18:30:55.315 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: p=2 2023-01-24T18:30:55.315 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:419: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2023-01-24T18:30:55.316 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: seq 1 50 2023-01-24T18:30:55.318 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:55.318 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2023-01-24T18:30:55.714 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:55.714 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2023-01-24T18:30:56.107 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:56.107 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2023-01-24T18:30:56.517 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:56.517 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2023-01-24T18:30:56.985 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:56.985 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2023-01-24T18:30:57.369 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:57.370 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2023-01-24T18:30:57.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:57.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2023-01-24T18:30:58.156 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:58.156 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2023-01-24T18:30:58.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:58.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2023-01-24T18:30:59.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:59.018 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2023-01-24T18:30:59.444 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:59.444 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2023-01-24T18:30:59.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:30:59.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2023-01-24T18:31:00.296 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:00.297 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2023-01-24T18:31:00.701 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:00.702 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2023-01-24T18:31:01.133 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:01.133 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2023-01-24T18:31:01.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:01.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2023-01-24T18:31:01.960 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:01.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2023-01-24T18:31:02.336 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:02.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2023-01-24T18:31:02.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:02.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2023-01-24T18:31:03.165 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:03.165 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2023-01-24T18:31:03.550 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:03.550 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2023-01-24T18:31:03.976 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:03.976 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2023-01-24T18:31:04.419 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:04.420 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2023-01-24T18:31:04.870 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:04.871 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2023-01-24T18:31:05.255 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:05.255 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2023-01-24T18:31:05.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:05.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2023-01-24T18:31:06.024 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:06.025 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2023-01-24T18:31:06.392 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:06.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2023-01-24T18:31:06.919 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:06.919 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2023-01-24T18:31:07.320 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:07.321 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2023-01-24T18:31:07.722 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:07.722 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2023-01-24T18:31:08.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:08.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2023-01-24T18:31:08.508 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:08.508 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2023-01-24T18:31:09.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:09.010 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2023-01-24T18:31:09.469 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:09.470 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2023-01-24T18:31:09.904 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:09.904 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2023-01-24T18:31:10.282 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:10.282 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2023-01-24T18:31:10.656 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:10.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2023-01-24T18:31:11.083 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:11.084 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2023-01-24T18:31:11.459 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:11.460 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2023-01-24T18:31:12.027 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:12.028 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2023-01-24T18:31:12.412 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:12.413 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2023-01-24T18:31:12.872 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:12.873 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2023-01-24T18:31:13.249 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:13.249 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2023-01-24T18:31:13.625 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:13.625 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2023-01-24T18:31:14.026 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:14.077 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2023-01-24T18:31:14.394 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:14.394 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2023-01-24T18:31:14.862 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:14.863 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2023-01-24T18:31:15.256 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:15.256 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2023-01-24T18:31:15.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2023-01-24T18:31:15.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2023-01-24T18:31:16.093 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: expr 2 + 1 2023-01-24T18:31:16.094 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: p=3 2023-01-24T18:31:16.096 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2023-01-24T18:31:16.096 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1227: get_not_primary: local poolname=test1 2023-01-24T18:31:16.096 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1228: get_not_primary: local objectname=obj1-p1 2023-01-24T18:31:16.097 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: get_primary test1 obj1-p1 2023-01-24T18:31:16.098 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: get_primary: local poolname=test1 2023-01-24T18:31:16.098 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: get_primary: local objectname=obj1-p1 2023-01-24T18:31:16.098 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: ceph --format json osd map test1 obj1-p1 2023-01-24T18:31:16.098 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: get_primary: jq .acting_primary 2023-01-24T18:31:16.520 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local primary=1 2023-01-24T18:31:16.522 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: get_not_primary: ceph --format json osd map test1 obj1-p1 2023-01-24T18:31:16.522 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2023-01-24T18:31:17.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: local otherosd=null 2023-01-24T18:31:17.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph pg dump pgs 2023-01-24T18:31:17.463 INFO:tasks.workunit.client.0.smithi082.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-24T18:31:17.463 INFO:tasks.workunit.client.0.smithi082.stdout:2.0 39 0 0 0 0 408944640 0 0 17 17 active+clean 2023-01-24T18:30:29.337998+0000 36'117 36:158 [0] 0 [0] 0 0'0 2023-01-24T18:30:08.213541+0000 0'0 2023-01-24T18:30:08.213541+0000 0 0 periodic scrub scheduled @ 2023-01-25T23:29:03.169679+0000 0 0 2023-01-24T18:31:17.463 INFO:tasks.workunit.client.0.smithi082.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2023-01-24T18:30:27.420623+0000 36'150 36:193 [1] 1 [1] 1 0'0 2023-01-24T18:30:04.698077+0000 0'0 2023-01-24T18:30:04.698077+0000 0 0 periodic scrub scheduled @ 2023-01-26T04:19:53.491176+0000 0 0 2023-01-24T18:31:17.463 INFO:tasks.workunit.client.0.smithi082.stdout: 2023-01-24T18:31:17.464 INFO:tasks.workunit.client.0.smithi082.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-24T18:31:17.466 INFO:tasks.workunit.client.0.smithi082.stderr:dumped pgs 2023-01-24T18:31:17.473 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ERRORS=0 2023-01-24T18:31:17.474 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2023-01-24T18:31:18.285 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 1 size to 2 2023-01-24T18:31:18.298 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2023-01-24T18:31:19.248 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 2 size to 2 2023-01-24T18:31:19.260 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: sleep 5 2023-01-24T18:31:24.264 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.1 2023-01-24T18:31:24.264 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:31:24.264 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:31:24.265 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:31:24.265 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:31:24.266 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:31:24.266 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:31:24.266 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2023-01-24T18:31:24.267 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:31:24.364 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.1 2023-01-24T18:31:24.365 INFO:tasks.workunit.client.0.smithi082.stdout:osd.1 2023-01-24T18:31:24.365 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2023-01-24T18:31:24.365 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:31:24.366 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:31:24.366 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:31:24.366 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:31:24.366 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:31:24.367 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:31:24.367 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:31:24.367 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:31:24.367 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 152, 2023-01-24T18:31:24.367 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:31:24.367 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:31:24.368 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:31:24.368 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:31:24.368 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:31:24.368 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:31:24.368 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:31:24.369 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:31:24.369 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:31:24.369 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:31:24.369 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "2.0", 2023-01-24T18:31:24.369 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 153, 2023-01-24T18:31:24.370 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:31:24.370 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:31:24.370 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:31:24.370 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:31:24.370 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:31:24.371 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: get_asok_path osd.0 2023-01-24T18:31:24.372 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-01-24T18:31:24.372 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-01-24T18:31:24.372 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:31:24.372 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:31:24.373 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:31:24.373 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.0.asok 2023-01-24T18:31:24.373 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: CEPH_ARGS= 2023-01-24T18:31:24.373 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.0.asok dump_recovery_reservations 2023-01-24T18:31:24.453 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:440: TEST_backfill_pool_priority: echo osd.0 2023-01-24T18:31:24.454 INFO:tasks.workunit.client.0.smithi082.stdout:osd.0 2023-01-24T18:31:24.454 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2023-01-24T18:31:24.454 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:31:24.455 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:31:24.455 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:31:24.455 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:31:24.455 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:31:24.455 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:31:24.456 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:31:24.456 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "2.0", 2023-01-24T18:31:24.456 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 153, 2023-01-24T18:31:24.456 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:31:24.457 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:31:24.457 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:31:24.457 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:31:24.457 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:31:24.458 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:31:24.458 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:31:24.458 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:31:24.458 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:31:24.458 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:31:24.459 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:31:24.459 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 152, 2023-01-24T18:31:24.459 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:31:24.459 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:31:24.459 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:31:24.459 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:31:24.460 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:31:24.460 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2023-01-24T18:31:24.460 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2023-01-24T18:31:24.498 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2023-01-24T18:31:24.498 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: ITEM=1.0 2023-01-24T18:31:24.498 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:445: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2023-01-24T18:31:24.499 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2023-01-24T18:31:24.499 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2023-01-24T18:31:24.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: PRIO=152 2023-01-24T18:31:24.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:451: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2023-01-24T18:31:24.541 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2023-01-24T18:31:24.541 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2023-01-24T18:31:24.582 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2023-01-24T18:31:24.582 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: ITEM=1.0 2023-01-24T18:31:24.582 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:460: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2023-01-24T18:31:24.583 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2023-01-24T18:31:24.584 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2023-01-24T18:31:24.625 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: PRIO=152 2023-01-24T18:31:24.625 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:466: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2023-01-24T18:31:24.626 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2023-01-24T18:31:24.627 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2023-01-24T18:31:24.667 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2023-01-24T18:31:24.668 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: ITEM=2.0 2023-01-24T18:31:24.668 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:475: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2023-01-24T18:31:24.669 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2023-01-24T18:31:24.669 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2023-01-24T18:31:24.711 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: PRIO=153 2023-01-24T18:31:24.711 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:481: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2023-01-24T18:31:24.712 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2023-01-24T18:31:24.713 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2023-01-24T18:31:24.756 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2023-01-24T18:31:24.756 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: ITEM=2.0 2023-01-24T18:31:24.756 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:490: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2023-01-24T18:31:24.757 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2023-01-24T18:31:24.757 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2023-01-24T18:31:24.799 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: PRIO=153 2023-01-24T18:31:24.800 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:496: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2023-01-24T18:31:24.800 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:503: TEST_backfill_pool_priority: wait_for_clean 2023-01-24T18:31:24.800 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-24T18:31:24.800 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-24T18:31:24.800 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-24T18:31:24.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-24T18:31:24.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-24T18:31:24.802 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-24T18:31:24.802 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-24T18:31:24.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-24T18:31:24.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-24T18:31:24.803 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-24T18:31:24.994 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-24T18:31:24.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-24T18:31:24.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-24T18:31:24.995 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:31:24.996 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:31:25.445 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:31:25.446 INFO:tasks.workunit.client.0.smithi082.stderr:1' 2023-01-24T18:31:25.446 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:31:25.446 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:31:25.447 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:31:25.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836505 2023-01-24T18:31:25.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836505 2023-01-24T18:31:25.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505' 2023-01-24T18:31:25.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:31:25.611 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:31:25.764 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672983 2023-01-24T18:31:25.765 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672983 2023-01-24T18:31:25.765 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2023-01-24T18:31:25.765 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:31:25.766 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836505 2023-01-24T18:31:25.766 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:31:25.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:31:25.770 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836505 2023-01-24T18:31:25.770 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:31:25.773 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836505 2023-01-24T18:31:25.773 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836505 2023-01-24T18:31:25.773 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2023-01-24T18:31:25.774 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:31:26.404 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836505 -lt 21474836505 2023-01-24T18:31:26.405 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:31:26.405 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672983 2023-01-24T18:31:26.406 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:31:26.407 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:31:26.409 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:31:26.409 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672983 2023-01-24T18:31:26.410 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672983 2023-01-24T18:31:26.410 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2023-01-24T18:31:26.411 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949672983 2023-01-24T18:31:26.411 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:31:26.955 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672981 -lt 42949672983 2023-01-24T18:31:26.956 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:31:27.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:31:27.958 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:31:28.375 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672983 -lt 42949672983 2023-01-24T18:31:28.376 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-24T18:31:28.377 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:28.377 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:29.067 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 2 == 0 2023-01-24T18:31:29.067 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:29.068 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:29.068 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:29.068 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:29.069 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:29.069 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:29.070 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:29.570 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:29.571 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:29.572 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:29.572 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:30.230 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:30.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' -1 2023-01-24T18:31:30.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: loop=0 2023-01-24T18:31:30.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: num_active_clean=0 2023-01-24T18:31:30.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:30.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:30.331 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:30.332 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:30.332 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:30.333 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:30.333 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:30.333 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:30.334 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:30.334 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:30.718 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:30.719 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:30.720 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:30.720 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:31.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:31.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:31.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:31.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:31.216 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:31.216 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:31.216 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:31.217 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:31.217 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:31.974 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=20806801 2023-01-24T18:31:31.974 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 20806801 '!=' null 2023-01-24T18:31:31.975 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:31.975 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:31.975 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:32.075 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:32.076 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:32.077 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:32.077 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:32.077 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:32.077 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:32.078 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:32.078 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:32.480 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:32.481 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:32.482 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:32.482 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:33.057 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:33.057 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:33.057 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:33.057 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:33.058 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:33.058 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:33.058 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:33.058 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:33.059 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:34.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=20806801 2023-01-24T18:31:34.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 20806801 '!=' null 2023-01-24T18:31:34.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:34.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:34.130 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:34.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:34.232 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:34.233 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:34.233 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:34.233 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:34.233 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:34.234 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:34.234 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:34.782 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:34.783 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:34.783 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:34.784 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:35.424 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:35.424 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:35.425 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:35.425 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:35.425 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:35.425 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:35.425 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:35.426 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:35.426 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:35.981 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=25332996 2023-01-24T18:31:35.981 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 25332996 '!=' null 2023-01-24T18:31:35.981 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:35.982 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:35.982 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:36.082 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:36.083 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:36.084 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:36.084 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:36.085 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:36.085 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:36.085 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:36.085 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:36.513 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:36.514 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:36.515 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:36.515 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:37.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:37.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:37.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:37.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:37.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:37.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:37.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:37.182 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:37.182 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:38.160 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=21839209 2023-01-24T18:31:38.160 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 21839209 '!=' null 2023-01-24T18:31:38.161 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:38.161 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:38.161 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:38.261 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:38.262 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:38.263 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:38.263 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:38.263 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:38.263 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:38.264 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:38.264 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:38.733 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:38.734 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:38.734 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:38.735 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:39.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:39.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:39.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:39.253 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:39.253 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:39.253 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:39.253 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:39.254 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:39.254 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:39.840 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=21839209 2023-01-24T18:31:39.840 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 21839209 '!=' null 2023-01-24T18:31:39.840 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:39.840 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:39.841 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:39.941 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:39.941 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:39.942 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:39.942 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:39.943 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:39.943 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:39.943 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:39.944 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:40.444 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:40.445 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:40.445 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:40.445 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:40.983 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:40.984 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:40.984 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:40.984 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:40.984 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:40.985 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:40.985 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:40.985 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:40.986 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:41.639 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=23586299 2023-01-24T18:31:41.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 23586299 '!=' null 2023-01-24T18:31:41.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:41.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:41.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:41.741 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:41.741 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:41.742 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:41.742 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:41.743 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:41.743 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:41.743 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:41.744 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:42.117 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:42.118 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:42.118 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:42.119 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:42.734 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:42.734 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:42.734 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:42.734 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:42.735 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:42.735 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:42.735 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:42.736 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:42.736 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:43.467 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=26206869 2023-01-24T18:31:43.467 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 26206869 '!=' null 2023-01-24T18:31:43.468 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:43.468 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:43.468 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:43.568 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:43.569 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:43.570 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:43.570 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:43.570 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:43.571 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:43.571 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:43.571 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:43.969 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:43.969 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:43.970 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:43.970 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:44.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:44.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:44.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:44.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:44.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:44.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:44.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:44.601 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:44.602 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:45.364 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=26206799 2023-01-24T18:31:45.364 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 26206799 '!=' null 2023-01-24T18:31:45.364 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:45.364 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:45.365 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:45.465 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:45.465 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:45.466 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:45.467 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:45.467 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:45.467 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:45.468 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:45.468 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:45.975 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:45.976 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:45.977 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:45.978 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:46.575 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:46.576 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:46.576 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:46.576 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:46.577 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:46.577 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:46.577 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:46.578 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:46.578 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:47.440 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=20965730 2023-01-24T18:31:47.441 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 20965730 '!=' null 2023-01-24T18:31:47.441 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:47.441 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:47.441 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:47.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:47.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:47.544 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:47.544 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:47.544 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:47.544 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:47.544 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:47.545 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:48.150 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:48.150 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:48.151 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:48.151 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:48.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:48.963 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:48.963 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:48.963 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:48.963 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:48.963 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:48.964 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:48.964 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:48.964 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:49.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=26207121 2023-01-24T18:31:49.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 26207121 '!=' null 2023-01-24T18:31:49.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:49.548 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:49.548 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:49.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:49.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:49.650 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:49.650 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:49.650 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:49.650 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:49.651 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:49.651 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:50.157 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:50.157 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:50.158 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:50.158 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:50.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:50.770 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:50.770 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:50.770 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:50.770 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:50.770 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:50.771 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:50.771 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:50.771 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:51.451 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=26207039 2023-01-24T18:31:51.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 26207039 '!=' null 2023-01-24T18:31:51.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:51.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:51.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:51.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:51.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:51.555 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:51.555 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:51.555 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:51.555 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:51.556 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:51.556 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:51.952 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:51.953 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:51.954 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:51.954 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:52.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:52.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:52.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:52.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:52.492 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:52.492 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:52.492 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:52.493 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:52.493 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:53.119 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=26207666 2023-01-24T18:31:53.120 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 26207666 '!=' null 2023-01-24T18:31:53.120 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:53.120 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:53.120 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:53.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:53.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:53.223 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:53.223 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:53.223 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:53.223 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:53.224 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:53.224 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:53.632 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:53.633 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:53.633 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:53.634 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:54.392 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:54.392 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:54.392 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:54.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:54.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:54.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:54.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:54.394 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:54.394 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:55.050 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=27081836 2023-01-24T18:31:55.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 27081836 '!=' null 2023-01-24T18:31:55.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:55.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:55.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:55.152 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:55.152 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:55.153 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:55.153 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:55.154 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:55.154 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:55.154 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:55.155 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:55.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:55.548 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:55.548 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:55.549 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:56.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:56.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:56.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:56.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:56.227 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:56.227 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:56.227 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:56.228 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:56.228 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:56.992 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=21840025 2023-01-24T18:31:56.992 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 21840025 '!=' null 2023-01-24T18:31:56.992 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:56.993 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:56.993 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:57.093 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:57.094 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:57.095 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:57.095 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:57.095 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:57.096 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:57.096 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:57.096 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:57.550 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:57.550 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:57.551 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:57.551 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:31:58.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:31:58.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:31:58.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:31:58.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:31:58.234 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:31:58.234 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:31:58.234 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:31:58.234 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:31:58.235 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:31:58.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=27081252 2023-01-24T18:31:58.990 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 27081252 '!=' null 2023-01-24T18:31:58.990 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:31:58.990 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:31:58.990 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:31:59.090 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:31:59.091 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:31:59.092 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:31:59.092 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:31:59.092 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:31:59.092 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:31:59.093 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:31:59.093 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:31:59.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:31:59.547 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:31:59.547 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:31:59.547 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:00.251 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:32:00.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:32:00.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:32:00.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:32:00.252 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:32:00.253 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:32:00.253 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:32:00.253 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:32:00.254 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:32:00.948 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=27081314 2023-01-24T18:32:00.949 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 27081314 '!=' null 2023-01-24T18:32:00.949 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:32:00.949 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:32:00.949 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:32:01.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:32:01.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:32:01.052 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:32:01.052 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:32:01.053 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:32:01.053 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:32:01.053 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:32:01.054 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:32:01.443 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:32:01.444 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:32:01.445 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:32:01.445 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:02.109 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:32:02.110 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:32:02.110 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:32:02.110 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:32:02.110 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:32:02.111 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:32:02.111 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:32:02.111 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:32:02.112 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:32:02.754 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=27255415 2023-01-24T18:32:02.754 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 27255415 '!=' null 2023-01-24T18:32:02.755 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:32:02.755 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:32:02.755 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:32:02.855 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:32:02.856 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:32:02.857 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:32:02.857 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:32:02.857 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:32:02.858 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:32:02.858 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:32:02.858 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:32:03.256 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:32:03.257 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:32:03.257 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:32:03.258 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:03.784 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 2 2023-01-24T18:32:03.784 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:32:03.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:32:03.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:32:03.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:32:03.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:32:03.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:32:03.786 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:32:03.786 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:32:04.298 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=26206475 2023-01-24T18:32:04.298 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 26206475 '!=' null 2023-01-24T18:32:04.298 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:32:04.299 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:32:04.299 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:32:04.399 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:32:04.400 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:32:04.401 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:32:04.401 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:32:04.401 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:32:04.401 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:32:04.402 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:32:04.402 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:32:04.787 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=1 2023-01-24T18:32:04.788 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:32:04.788 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:04.788 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:32:05.311 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 1 = 2 2023-01-24T18:32:05.311 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 1 '!=' 0 2023-01-24T18:32:05.311 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: loop=0 2023-01-24T18:32:05.312 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: num_active_clean=1 2023-01-24T18:32:05.312 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:32:05.312 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:32:05.412 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:32:05.412 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:32:05.413 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:32:05.413 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:32:05.414 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:32:05.414 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:32:05.414 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:32:05.415 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:32:05.852 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=1 2023-01-24T18:32:05.853 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:32:05.854 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:32:05.854 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:06.384 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 1 = 2 2023-01-24T18:32:06.385 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 1 '!=' 1 2023-01-24T18:32:06.385 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:32:06.385 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:32:06.385 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:32:06.385 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:32:06.386 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:32:06.386 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:32:06.386 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:32:06.907 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=24884271 2023-01-24T18:32:06.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 24884271 '!=' null 2023-01-24T18:32:06.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:32:06.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:32:06.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:32:07.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:32:07.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:32:07.010 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:32:07.010 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:32:07.011 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:32:07.011 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:32:07.012 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:32:07.012 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:32:07.408 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=1 2023-01-24T18:32:07.409 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:32:07.410 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:32:07.410 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:07.987 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 1 = 2 2023-01-24T18:32:07.987 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 1 '!=' 1 2023-01-24T18:32:07.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:32:07.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:32:07.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:32:07.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:32:07.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:32:07.989 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:32:07.989 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:32:08.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=5241383 2023-01-24T18:32:08.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 5241383 '!=' null 2023-01-24T18:32:08.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:32:08.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 2023-01-24T18:32:08.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:32:08.633 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:32:08.633 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:32:08.634 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:32:08.635 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:32:08.635 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:32:08.635 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:32:08.636 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:32:08.636 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:32:09.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=2 2023-01-24T18:32:09.018 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:32:09.018 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:32:09.019 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:32:09.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 2 = 2 2023-01-24T18:32:09.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-24T18:32:09.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-24T18:32:09.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:505: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2023-01-24T18:32:09.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: echo TEST PASSED 2023-01-24T18:32:09.542 INFO:tasks.workunit.client.0.smithi082.stdout:TEST PASSED 2023-01-24T18:32:09.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: delete_pool test1 2023-01-24T18:32:09.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2023-01-24T18:32:09.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2023-01-24T18:32:10.112 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test1' does not exist 2023-01-24T18:32:10.120 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:513: TEST_backfill_pool_priority: delete_pool test2 2023-01-24T18:32:10.121 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2023-01-24T18:32:10.121 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2023-01-24T18:32:10.715 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test2' does not exist 2023-01-24T18:32:10.724 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:514: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2023-01-24T18:32:10.726 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-24T18:32:10.726 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-24T18:32:10.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-24T18:32:10.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-24T18:32:10.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-24T18:32:16.049 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-24T18:32:16.049 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:515: TEST_backfill_pool_priority: return 0 2023-01-24T18:32:16.049 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:43: run: teardown td/osd-backfill-prio 2023-01-24T18:32:16.050 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2023-01-24T18:32:16.050 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-24T18:32:16.050 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2023-01-24T18:32:16.051 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-24T18:32:16.051 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-24T18:32:16.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-24T18:32:16.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-24T18:32:16.052 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-24T18:32:16.061 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-24T18:32:16.062 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-24T18:32:16.063 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-24T18:32:16.064 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-24T18:32:16.066 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-24T18:32:16.066 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-24T18:32:16.067 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-24T18:32:16.068 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:32:16.068 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-24T18:32:16.068 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-24T18:32:16.069 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:32:16.070 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-24T18:32:16.072 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-24T18:32:16.072 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2023-01-24T18:32:16.089 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-24T18:32:16.090 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:16.090 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:16.090 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.23029 2023-01-24T18:32:16.091 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-24T18:32:16.092 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-24T18:32:16.092 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:40: run: for func in $funcs 2023-01-24T18:32:16.092 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:41: run: setup td/osd-backfill-prio 2023-01-24T18:32:16.092 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2023-01-24T18:32:16.093 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2023-01-24T18:32:16.093 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2023-01-24T18:32:16.093 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-24T18:32:16.093 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2023-01-24T18:32:16.094 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-24T18:32:16.094 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-24T18:32:16.095 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-24T18:32:16.095 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-24T18:32:16.095 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-24T18:32:16.097 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-24T18:32:16.097 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-24T18:32:16.098 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-24T18:32:16.099 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-24T18:32:16.100 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-24T18:32:16.100 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-24T18:32:16.101 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-24T18:32:16.103 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:32:16.103 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-24T18:32:16.103 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-24T18:32:16.104 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T18:32:16.105 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-24T18:32:16.107 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-24T18:32:16.107 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2023-01-24T18:32:16.109 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-24T18:32:16.109 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:16.110 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:16.110 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.23029 2023-01-24T18:32:16.111 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-24T18:32:16.112 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-24T18:32:16.112 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2023-01-24T18:32:16.114 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-24T18:32:16.114 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:16.115 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:16.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.23029 2023-01-24T18:32:16.117 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-24T18:32:16.118 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-01-24T18:32:16.118 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-24T18:32:16.118 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2023-01-24T18:32:16.118 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: TEST_backfill_priority td/osd-backfill-prio 2023-01-24T18:32:16.119 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:49: TEST_backfill_priority: local dir=td/osd-backfill-prio 2023-01-24T18:32:16.119 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:50: TEST_backfill_priority: local pools=10 2023-01-24T18:32:16.119 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:51: TEST_backfill_priority: local OSDS=5 2023-01-24T18:32:16.120 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:53: TEST_backfill_priority: expr 150 + 1 2023-01-24T18:32:16.121 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:53: TEST_backfill_priority: local degraded_prio=151 2023-01-24T18:32:16.121 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:54: TEST_backfill_priority: local max_tries=10 2023-01-24T18:32:16.121 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:56: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2023-01-24T18:32:16.121 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2023-01-24T18:32:16.122 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-24T18:32:16.122 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-24T18:32:16.122 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-24T18:32:16.122 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2023-01-24T18:32:16.122 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2023-01-24T18:32:16.517 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-24T18:32:16.518 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:32:16.518 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:32:16.519 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:32:16.519 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:16.519 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:16.519 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:16.520 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-01-24T18:32:16.815 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-24T18:32:16.817 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-24T18:32:16.817 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-24T18:32:16.817 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-24T18:32:16.818 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-24T18:32:16.818 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-24T18:32:16.819 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-24T18:32:16.819 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-24T18:32:16.819 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-24T18:32:16.820 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:32:16.820 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:16.820 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:16.821 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-mon.a.asok 2023-01-24T18:32:16.821 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-24T18:32:16.821 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.23029/ceph-mon.a.asok config get fsid 2023-01-24T18:32:16.913 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-24T18:32:16.914 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-24T18:32:16.914 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-24T18:32:16.914 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-24T18:32:16.915 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-24T18:32:16.916 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-24T18:32:16.916 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-24T18:32:16.916 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-24T18:32:16.916 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:32:16.917 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:16.917 INFO:tasks.workunit.client.0.smithi082.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:16.917 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-mon.a.asok 2023-01-24T18:32:16.917 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-24T18:32:16.918 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.23029/ceph-mon.a.asok config get mon_host 2023-01-24T18:32:16.996 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:57: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2023-01-24T18:32:16.996 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-prio 2023-01-24T18:32:16.996 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-24T18:32:16.997 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-24T18:32:16.997 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-24T18:32:16.997 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-prio/x 2023-01-24T18:32:16.997 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-24T18:32:17.913 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-24T18:32:17.914 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:32:17.914 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:32:17.915 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:32:17.915 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:17.915 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:17.915 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:17.916 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-24T18:32:17.917 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-24T18:32:17.953 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:58: TEST_backfill_priority: export CEPH_ARGS 2023-01-24T18:32:17.955 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: expr 5 - 1 2023-01-24T18:32:17.956 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: seq 0 4 2023-01-24T18:32:17.959 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:32:17.959 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2023-01-24T18:32:17.960 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:32:17.960 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:32:17.960 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-24T18:32:17.960 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:32:17.960 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/0 2023-01-24T18:32:17.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:32:17.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:32:17.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:32:17.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:32:17.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2023-01-24T18:32:17.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2023-01-24T18:32:17.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:32:17.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:32:17.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:32:17.962 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:32:17.963 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:32:17.963 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:32:17.963 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:32:17.963 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:17.963 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:17.965 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:17.965 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:17.965 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:32:17.966 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:32:17.966 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:32:17.966 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:32:17.966 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:32:17.966 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:32:17.967 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:32:17.967 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:17.967 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:32:17.967 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:32:17.967 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/0 2023-01-24T18:32:17.968 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:32:17.968 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=73fbbaba-b21b-4eb9-bf00-3d42c0b71005 2023-01-24T18:32:17.968 INFO:tasks.workunit.client.0.smithi082.stdout:add osd0 73fbbaba-b21b-4eb9-bf00-3d42c0b71005 2023-01-24T18:32:17.969 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 73fbbaba-b21b-4eb9-bf00-3d42c0b71005' 2023-01-24T18:32:17.969 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:32:17.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAxJNBjW07eOhAA06421bz9yE/nWPbybdSoow== 2023-01-24T18:32:17.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAxJNBjW07eOhAA06421bz9yE/nWPbybdSoow=="}' 2023-01-24T18:32:17.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 73fbbaba-b21b-4eb9-bf00-3d42c0b71005 -i td/osd-backfill-prio/0/new.json 2023-01-24T18:32:18.334 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:32:18.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/0/new.json 2023-01-24T18:32:18.349 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAxJNBjW07eOhAA06421bz9yE/nWPbybdSoow== --osd-uuid 73fbbaba-b21b-4eb9-bf00-3d42c0b71005 2023-01-24T18:32:18.382 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.377+0000 7fa48c9903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:18.385 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.381+0000 7fa48c9903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:18.387 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.381+0000 7fa48c9903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:18.388 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.381+0000 7fa48c9903c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2023-01-24T18:32:18.388 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.381+0000 7fa48c9903c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2023-01-24T18:32:18.388 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.381+0000 7fa48c9903c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2023-01-24T18:32:18.472 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:18.469+0000 7fa48c9903c0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2023-01-24T18:32:24.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2023-01-24T18:32:24.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:32:24.700 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd0 key to auth repository 2023-01-24T18:32:24.700 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-24T18:32:24.700 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:32:25.275 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.0 2023-01-24T18:32:25.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-24T18:32:25.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:32:25.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:25.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:32:25.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:32:25.312 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:25.305+0000 7f63e40a93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:25.315 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:25.309+0000 7f63e40a93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:25.317 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:25.313+0000 7f63e40a93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:25.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-24T18:32:25.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:32:25.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-24T18:32:25.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:32:25.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:32:25.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:25.728 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:32:25.728 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:32:25.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:25.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:32:25.875 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:25.869+0000 7f63e40a93c0 -1 Falling back to public interface 2023-01-24T18:32:26.228 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:27.229 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:27.230 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:27.230 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:32:27.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:32:27.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:27.231 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:32:27.627 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:28.442 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:28.437+0000 7f63e40a93c0 -1 osd.0 0 log_to_monitors true 2023-01-24T18:32:28.629 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:28.629 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:28.630 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:32:28.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:32:28.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:28.631 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:32:29.101 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:30.102 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:30.103 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:30.103 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:32:30.104 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:32:30.104 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:30.104 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-24T18:32:30.520 INFO:tasks.workunit.client.0.smithi082.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/32670,v1:127.0.0.1:6803/32670] [v2:127.0.0.1:6804/32670,v1:127.0.0.1:6805/32670] exists,up 73fbbaba-b21b-4eb9-bf00-3d42c0b71005 2023-01-24T18:32:30.523 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:32:30.523 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:32:30.523 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:32:30.524 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:32:30.524 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2023-01-24T18:32:30.524 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:32:30.524 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:32:30.524 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-24T18:32:30.525 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:32:30.525 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/1 2023-01-24T18:32:30.525 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:32:30.525 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:32:30.525 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:32:30.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:32:30.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2023-01-24T18:32:30.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2023-01-24T18:32:30.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:32:30.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:32:30.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:32:30.527 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:32:30.527 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:32:30.527 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:32:30.529 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:32:30.529 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:30.529 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:30.530 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:30.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:30.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:32:30.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:32:30.530 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:32:30.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:32:30.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:32:30.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:32:30.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:32:30.531 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:30.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:32:30.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:32:30.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/1 2023-01-24T18:32:30.532 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:32:30.532 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f2dde22d-98a1-471d-b773-ff026a5ddf67 2023-01-24T18:32:30.533 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 f2dde22d-98a1-471d-b773-ff026a5ddf67' 2023-01-24T18:32:30.533 INFO:tasks.workunit.client.0.smithi082.stdout:add osd1 f2dde22d-98a1-471d-b773-ff026a5ddf67 2023-01-24T18:32:30.533 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:32:30.549 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA+JNBjOtayIBAAcBZ8R3xO8n/aorKKAD0MaQ== 2023-01-24T18:32:30.549 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA+JNBjOtayIBAAcBZ8R3xO8n/aorKKAD0MaQ=="}' 2023-01-24T18:32:30.550 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f2dde22d-98a1-471d-b773-ff026a5ddf67 -i td/osd-backfill-prio/1/new.json 2023-01-24T18:32:31.081 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:32:31.093 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/1/new.json 2023-01-24T18:32:31.095 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA+JNBjOtayIBAAcBZ8R3xO8n/aorKKAD0MaQ== --osd-uuid f2dde22d-98a1-471d-b773-ff026a5ddf67 2023-01-24T18:32:31.132 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.125+0000 7feab34473c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:31.164 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.161+0000 7feab34473c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:31.166 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.161+0000 7feab34473c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:31.167 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.161+0000 7feab34473c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2023-01-24T18:32:31.167 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.161+0000 7feab34473c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2023-01-24T18:32:31.167 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.161+0000 7feab34473c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2023-01-24T18:32:31.255 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:31.253+0000 7feab34473c0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2023-01-24T18:32:37.281 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2023-01-24T18:32:37.281 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:32:37.282 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-24T18:32:37.283 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd1 key to auth repository 2023-01-24T18:32:37.283 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:32:37.837 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-24T18:32:37.838 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.1 2023-01-24T18:32:37.838 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:37.839 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:32:37.839 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:32:37.839 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:32:37.873 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:37.869+0000 7f7d6f9453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:37.876 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:37.869+0000 7f7d6f9453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:37.877 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:37.873+0000 7f7d6f9453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:38.289 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-24T18:32:38.290 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:32:38.290 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-24T18:32:38.290 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:32:38.290 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:32:38.291 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:38.291 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:32:38.291 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:32:38.292 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:38.292 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:32:38.434 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:38.429+0000 7f7d6f9453c0 -1 Falling back to public interface 2023-01-24T18:32:38.782 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:39.784 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:39.784 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:39.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:32:39.785 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:32:39.785 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:39.786 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:32:40.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:41.014 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:41.009+0000 7f7d6f9453c0 -1 osd.1 0 log_to_monitors true 2023-01-24T18:32:41.228 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:41.229 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:41.229 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:32:41.230 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:32:41.230 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:41.230 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:32:41.753 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:42.755 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:42.756 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:42.756 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:32:42.757 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:32:42.757 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:42.757 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-24T18:32:43.171 INFO:tasks.workunit.client.0.smithi082.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34076,v1:127.0.0.1:6811/34076] [v2:127.0.0.1:6812/34076,v1:127.0.0.1:6813/34076] exists,up f2dde22d-98a1-471d-b773-ff026a5ddf67 2023-01-24T18:32:43.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:32:43.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:32:43.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:32:43.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:32:43.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2023-01-24T18:32:43.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:32:43.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:32:43.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-24T18:32:43.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:32:43.176 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/2 2023-01-24T18:32:43.176 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:32:43.176 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:32:43.176 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:32:43.176 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:32:43.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2023-01-24T18:32:43.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2023-01-24T18:32:43.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:32:43.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:32:43.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:32:43.178 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:32:43.178 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:32:43.178 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:32:43.178 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:32:43.178 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:43.178 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:43.179 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:43.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:43.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:32:43.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:32:43.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:32:43.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:32:43.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:32:43.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:32:43.182 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:32:43.182 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:43.182 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:32:43.182 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:32:43.183 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/2 2023-01-24T18:32:43.183 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:32:43.183 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9fbbc2d8-6bbe-4a27-a4bb-3c440b6f7b0b 2023-01-24T18:32:43.183 INFO:tasks.workunit.client.0.smithi082.stdout:add osd2 9fbbc2d8-6bbe-4a27-a4bb-3c440b6f7b0b 2023-01-24T18:32:43.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 9fbbc2d8-6bbe-4a27-a4bb-3c440b6f7b0b' 2023-01-24T18:32:43.184 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:32:43.198 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBLJNBjh9O9CxAAt6A7dxp+bHUPrHP8RlmNmA== 2023-01-24T18:32:43.198 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBLJNBjh9O9CxAAt6A7dxp+bHUPrHP8RlmNmA=="}' 2023-01-24T18:32:43.198 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9fbbc2d8-6bbe-4a27-a4bb-3c440b6f7b0b -i td/osd-backfill-prio/2/new.json 2023-01-24T18:32:43.773 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:32:43.787 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/2/new.json 2023-01-24T18:32:43.788 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBLJNBjh9O9CxAAt6A7dxp+bHUPrHP8RlmNmA== --osd-uuid 9fbbc2d8-6bbe-4a27-a4bb-3c440b6f7b0b 2023-01-24T18:32:43.828 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.825+0000 7fa18ff8a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:43.870 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.865+0000 7fa18ff8a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:43.871 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.865+0000 7fa18ff8a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:43.872 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.865+0000 7fa18ff8a3c0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2023-01-24T18:32:43.872 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.865+0000 7fa18ff8a3c0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2023-01-24T18:32:43.873 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.865+0000 7fa18ff8a3c0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2023-01-24T18:32:43.953 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:43.949+0000 7fa18ff8a3c0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2023-01-24T18:32:50.520 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2023-01-24T18:32:50.521 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:32:50.522 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-24T18:32:50.523 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd2 key to auth repository 2023-01-24T18:32:50.523 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:32:51.085 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.2 2023-01-24T18:32:51.086 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-24T18:32:51.086 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:32:51.087 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:51.087 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:32:51.087 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:32:51.121 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:51.116+0000 7f82f16453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:51.170 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:51.164+0000 7f82f16453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:51.171 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:51.164+0000 7f82f16453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:51.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-24T18:32:51.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:32:51.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-24T18:32:51.542 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:32:51.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:32:51.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:51.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:32:51.543 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:32:51.544 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:51.544 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-24T18:32:51.718 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:51.712+0000 7f82f16453c0 -1 Falling back to public interface 2023-01-24T18:32:51.936 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:52.938 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:52.938 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:52.939 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:32:52.939 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:32:52.939 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-24T18:32:52.940 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:53.352 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:54.354 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:54.354 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:54.354 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:32:54.355 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:32:54.355 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:54.355 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-24T18:32:54.380 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:54.372+0000 7f82f16453c0 -1 osd.2 0 log_to_monitors true 2023-01-24T18:32:54.790 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:32:55.792 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:32:55.792 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:32:55.792 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:32:55.793 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:32:55.793 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:32:55.793 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-24T18:32:56.199 INFO:tasks.workunit.client.0.smithi082.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35484,v1:127.0.0.1:6819/35484] [v2:127.0.0.1:6820/35484,v1:127.0.0.1:6821/35484] exists,up 9fbbc2d8-6bbe-4a27-a4bb-3c440b6f7b0b 2023-01-24T18:32:56.203 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:32:56.203 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:32:56.204 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:32:56.204 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:32:56.204 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2023-01-24T18:32:56.204 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:32:56.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:32:56.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-01-24T18:32:56.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:32:56.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/3 2023-01-24T18:32:56.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:32:56.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:32:56.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:32:56.207 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:32:56.207 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2023-01-24T18:32:56.207 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2023-01-24T18:32:56.208 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:32:56.208 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:32:56.208 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:32:56.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:32:56.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:32:56.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:32:56.210 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:32:56.210 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:32:56.211 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:32:56.211 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:56.211 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:32:56.212 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:32:56.212 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:32:56.212 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:32:56.213 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:32:56.213 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:32:56.213 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:32:56.214 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:32:56.214 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:32:56.214 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:32:56.214 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:32:56.214 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/3 2023-01-24T18:32:56.215 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:32:56.215 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=edd66063-53a7-4a2d-aa65-6299cf0207b6 2023-01-24T18:32:56.216 INFO:tasks.workunit.client.0.smithi082.stdout:add osd3 edd66063-53a7-4a2d-aa65-6299cf0207b6 2023-01-24T18:32:56.216 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 edd66063-53a7-4a2d-aa65-6299cf0207b6' 2023-01-24T18:32:56.216 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:32:56.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBYJNBj44fSDRAARQXikQKOqrrfSEnAD8A6+Q== 2023-01-24T18:32:56.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBYJNBj44fSDRAARQXikQKOqrrfSEnAD8A6+Q=="}' 2023-01-24T18:32:56.234 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new edd66063-53a7-4a2d-aa65-6299cf0207b6 -i td/osd-backfill-prio/3/new.json 2023-01-24T18:32:56.763 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:32:56.780 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/3/new.json 2023-01-24T18:32:56.782 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBYJNBj44fSDRAARQXikQKOqrrfSEnAD8A6+Q== --osd-uuid edd66063-53a7-4a2d-aa65-6299cf0207b6 2023-01-24T18:32:56.821 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.816+0000 7f4959c8b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:56.845 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.840+0000 7f4959c8b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:56.846 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.840+0000 7f4959c8b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:32:56.847 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.840+0000 7f4959c8b3c0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2023-01-24T18:32:56.848 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.840+0000 7f4959c8b3c0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2023-01-24T18:32:56.848 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.840+0000 7f4959c8b3c0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2023-01-24T18:32:56.935 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:32:56.928+0000 7f4959c8b3c0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2023-01-24T18:33:03.075 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2023-01-24T18:33:03.075 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:33:03.077 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2023-01-24T18:33:03.078 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd3 key to auth repository 2023-01-24T18:33:03.078 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:33:03.626 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.3 2023-01-24T18:33:03.627 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2023-01-24T18:33:03.627 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:33:03.627 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:33:03.627 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:33:03.628 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:33:03.656 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:03.648+0000 7fcffe5113c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:03.658 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:03.652+0000 7fcffe5113c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:03.665 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:03.652+0000 7fcffe5113c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:04.061 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2023-01-24T18:33:04.061 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:33:04.062 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=3 2023-01-24T18:33:04.062 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:33:04.062 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:33:04.062 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:04.063 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:33:04.063 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:33:04.064 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:04.064 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-24T18:33:04.215 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:04.208+0000 7fcffe5113c0 -1 Falling back to public interface 2023-01-24T18:33:04.547 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:33:05.550 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:33:05.550 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:05.551 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:33:05.552 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:33:05.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:05.553 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-24T18:33:06.007 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:33:06.765 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:06.760+0000 7fcffe5113c0 -1 osd.3 0 log_to_monitors true 2023-01-24T18:33:07.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:33:07.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:07.010 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:33:07.011 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:33:07.011 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:07.012 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-24T18:33:07.448 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:33:08.450 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:33:08.450 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:08.451 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:33:08.451 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:33:08.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:08.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-24T18:33:08.855 INFO:tasks.workunit.client.0.smithi082.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/36888,v1:127.0.0.1:6827/36888] [v2:127.0.0.1:6828/36888,v1:127.0.0.1:6829/36888] exists,up edd66063-53a7-4a2d-aa65-6299cf0207b6 2023-01-24T18:33:08.857 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:33:08.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:33:08.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:33:08.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-01-24T18:33:08.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2023-01-24T18:33:08.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2023-01-24T18:33:08.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-24T18:33:08.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-01-24T18:33:08.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-24T18:33:08.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/4 2023-01-24T18:33:08.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2023-01-24T18:33:08.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-24T18:33:08.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-24T18:33:08.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-24T18:33:08.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2023-01-24T18:33:08.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2023-01-24T18:33:08.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-24T18:33:08.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-24T18:33:08.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2023-01-24T18:33:08.861 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-24T18:33:08.861 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-24T18:33:08.861 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-24T18:33:08.862 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-24T18:33:08.862 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:33:08.862 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:33:08.862 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:33:08.864 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' 2023-01-24T18:33:08.864 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-24T18:33:08.864 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-24T18:33:08.864 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-24T18:33:08.864 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2023-01-24T18:33:08.865 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2023-01-24T18:33:08.865 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-24T18:33:08.865 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-24T18:33:08.865 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:33:08.865 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-24T18:33:08.866 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-24T18:33:08.866 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/4 2023-01-24T18:33:08.866 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-24T18:33:08.866 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=52097762-87f3-4ca0-a9f1-0679ac852f17 2023-01-24T18:33:08.866 INFO:tasks.workunit.client.0.smithi082.stdout:add osd4 52097762-87f3-4ca0-a9f1-0679ac852f17 2023-01-24T18:33:08.867 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 52097762-87f3-4ca0-a9f1-0679ac852f17' 2023-01-24T18:33:08.867 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-24T18:33:08.889 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBkJNBjbt/rNBAAFwXS41UUccM1QlYs65esRg== 2023-01-24T18:33:08.889 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBkJNBjbt/rNBAAFwXS41UUccM1QlYs65esRg=="}' 2023-01-24T18:33:08.889 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 52097762-87f3-4ca0-a9f1-0679ac852f17 -i td/osd-backfill-prio/4/new.json 2023-01-24T18:33:09.522 INFO:tasks.workunit.client.0.smithi082.stdout:4 2023-01-24T18:33:09.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/4/new.json 2023-01-24T18:33:09.540 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBkJNBjbt/rNBAAFwXS41UUccM1QlYs65esRg== --osd-uuid 52097762-87f3-4ca0-a9f1-0679ac852f17 2023-01-24T18:33:09.579 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.576+0000 7f07aef123c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:09.581 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.580+0000 7f07aef123c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:09.582 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.580+0000 7f07aef123c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:09.582 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.580+0000 7f07aef123c0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2023-01-24T18:33:09.583 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.580+0000 7f07aef123c0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2023-01-24T18:33:09.583 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.580+0000 7f07aef123c0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2023-01-24T18:33:09.696 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:09.692+0000 7f07aef123c0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2023-01-24T18:33:15.850 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2023-01-24T18:33:15.850 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-24T18:33:15.852 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2023-01-24T18:33:15.853 INFO:tasks.workunit.client.0.smithi082.stdout:adding osd4 key to auth repository 2023-01-24T18:33:15.853 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-24T18:33:16.387 INFO:tasks.workunit.client.0.smithi082.stdout:start osd.4 2023-01-24T18:33:16.388 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2023-01-24T18:33:16.388 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=99add1ad-305f-441d-b84f-cbc60f064788 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.23029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-24T18:33:16.388 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-24T18:33:16.388 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-24T18:33:16.389 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-24T18:33:16.424 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:16.420+0000 7fdf31be93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:16.446 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:16.444+0000 7fdf31be93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:16.447 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:16.444+0000 7fdf31be93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-24T18:33:16.823 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2023-01-24T18:33:16.823 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-24T18:33:16.824 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=4 2023-01-24T18:33:16.824 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-24T18:33:16.824 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-24T18:33:16.824 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:16.824 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-24T18:33:16.825 INFO:tasks.workunit.client.0.smithi082.stdout:0 2023-01-24T18:33:16.825 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:16.825 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-24T18:33:17.001 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:16.996+0000 7fdf31be93c0 -1 Falling back to public interface 2023-01-24T18:33:17.232 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:33:18.234 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:33:18.234 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:18.235 INFO:tasks.workunit.client.0.smithi082.stdout:1 2023-01-24T18:33:18.235 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-24T18:33:18.236 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:18.236 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-24T18:33:18.626 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:33:19.550 INFO:tasks.workunit.client.0.smithi082.stderr:2023-01-24T18:33:19.548+0000 7fdf31be93c0 -1 osd.4 0 log_to_monitors true 2023-01-24T18:33:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:33:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:19.628 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-24T18:33:19.629 INFO:tasks.workunit.client.0.smithi082.stdout:2 2023-01-24T18:33:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:19.629 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-24T18:33:20.075 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-24T18:33:21.077 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-24T18:33:21.077 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-24T18:33:21.078 INFO:tasks.workunit.client.0.smithi082.stdout:3 2023-01-24T18:33:21.078 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-24T18:33:21.079 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-24T18:33:21.079 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-24T18:33:21.494 INFO:tasks.workunit.client.0.smithi082.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/38295,v1:127.0.0.1:6835/38295] [v2:127.0.0.1:6836/38295,v1:127.0.0.1:6837/38295] exists,up 52097762-87f3-4ca0-a9f1-0679ac852f17 2023-01-24T18:33:21.494 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-24T18:33:21.495 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-24T18:33:21.495 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-24T18:33:21.495 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: seq 1 10 2023-01-24T18:33:21.497 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:21.497 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test1 1 1 2023-01-24T18:33:21.497 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2023-01-24T18:33:22.217 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test1' already exists 2023-01-24T18:33:22.230 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:23.232 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test1 size 2 2023-01-24T18:33:24.166 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 1 size to 2 2023-01-24T18:33:24.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:24.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test2 1 1 2023-01-24T18:33:24.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2023-01-24T18:33:24.871 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test2' already exists 2023-01-24T18:33:24.879 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:25.881 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test2 size 2 2023-01-24T18:33:26.839 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 2 size to 2 2023-01-24T18:33:26.851 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:26.851 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test3 1 1 2023-01-24T18:33:26.851 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2023-01-24T18:33:27.584 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test3' already exists 2023-01-24T18:33:27.592 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:28.594 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test3 size 2 2023-01-24T18:33:29.654 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 3 size to 2 2023-01-24T18:33:29.667 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:29.667 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test4 1 1 2023-01-24T18:33:29.667 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2023-01-24T18:33:30.382 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test4' already exists 2023-01-24T18:33:30.394 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:31.397 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test4 size 2 2023-01-24T18:33:32.292 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 4 size to 2 2023-01-24T18:33:32.304 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:32.304 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test5 1 1 2023-01-24T18:33:32.304 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2023-01-24T18:33:33.180 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test5' already exists 2023-01-24T18:33:33.192 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:34.194 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test5 size 2 2023-01-24T18:33:35.158 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 5 size to 2 2023-01-24T18:33:35.173 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:35.173 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test6 1 1 2023-01-24T18:33:35.173 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2023-01-24T18:33:35.947 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test6' already exists 2023-01-24T18:33:35.955 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:36.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test6 size 2 2023-01-24T18:33:37.840 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 6 size to 2 2023-01-24T18:33:37.854 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:37.854 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test7 1 1 2023-01-24T18:33:37.855 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2023-01-24T18:33:38.439 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test7' already exists 2023-01-24T18:33:38.452 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:39.454 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test7 size 2 2023-01-24T18:33:40.448 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 7 size to 2 2023-01-24T18:33:40.459 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:40.459 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test8 1 1 2023-01-24T18:33:40.460 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2023-01-24T18:33:41.168 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test8' already exists 2023-01-24T18:33:41.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:42.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test8 size 2 2023-01-24T18:33:43.170 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 8 size to 2 2023-01-24T18:33:43.183 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:43.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test9 1 1 2023-01-24T18:33:43.184 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2023-01-24T18:33:44.201 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test9' already exists 2023-01-24T18:33:44.210 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:45.212 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test9 size 2 2023-01-24T18:33:46.629 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 9 size to 2 2023-01-24T18:33:46.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:33:46.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test10 1 1 2023-01-24T18:33:46.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2023-01-24T18:33:47.684 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test10' already exists 2023-01-24T18:33:47.697 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-24T18:33:48.699 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test10 size 2 2023-01-24T18:33:49.718 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 10 size to 2 2023-01-24T18:33:49.731 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:70: TEST_backfill_priority: sleep 5 2023-01-24T18:33:54.733 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:72: TEST_backfill_priority: wait_for_clean 2023-01-24T18:33:54.733 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-24T18:33:54.733 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-24T18:33:54.734 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-24T18:33:54.734 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-24T18:33:54.734 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-24T18:33:54.735 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-24T18:33:54.735 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-24T18:33:54.735 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-24T18:33:54.736 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-24T18:33:54.736 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-24T18:33:54.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-24T18:33:54.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-24T18:33:54.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-24T18:33:54.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:33:54.958 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:33:55.375 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:33:55.375 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:33:55.375 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:33:55.376 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:33:55.376 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:33:55.376 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:33:55.376 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:33:55.377 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:33:55.545 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836500 2023-01-24T18:33:55.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836500 2023-01-24T18:33:55.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836500' 2023-01-24T18:33:55.546 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:33:55.546 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:33:55.697 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672977 2023-01-24T18:33:55.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672977 2023-01-24T18:33:55.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2023-01-24T18:33:55.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:33:55.698 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:33:55.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509455 2023-01-24T18:33:55.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509455 2023-01-24T18:33:55.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455' 2023-01-24T18:33:55.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:33:55.862 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:33:56.013 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345933 2023-01-24T18:33:56.014 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345933 2023-01-24T18:33:56.014 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933' 2023-01-24T18:33:56.014 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:33:56.015 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:33:56.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182410 2023-01-24T18:33:56.174 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182410 2023-01-24T18:33:56.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345933 4-107374182410' 2023-01-24T18:33:56.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:33:56.175 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836500 2023-01-24T18:33:56.176 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:33:56.176 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:33:56.177 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836500 2023-01-24T18:33:56.178 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:33:56.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836500 2023-01-24T18:33:56.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2023-01-24T18:33:56.180 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836500 2023-01-24T18:33:56.180 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:33:56.593 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836498 -lt 21474836500 2023-01-24T18:33:56.594 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:33:57.595 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:33:57.596 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:33:58.014 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836500 -lt 21474836500 2023-01-24T18:33:58.014 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:33:58.015 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672977 2023-01-24T18:33:58.015 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:33:58.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:33:58.018 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672977 2023-01-24T18:33:58.018 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:33:58.020 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672977 2023-01-24T18:33:58.020 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2023-01-24T18:33:58.020 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949672977 2023-01-24T18:33:58.021 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:33:58.395 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672977 -lt 42949672977 2023-01-24T18:33:58.396 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:33:58.396 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509455 2023-01-24T18:33:58.397 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:33:58.398 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:33:58.400 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509455 2023-01-24T18:33:58.400 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:33:58.401 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509455 2023-01-24T18:33:58.401 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2023-01-24T18:33:58.402 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509455 2023-01-24T18:33:58.402 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:33:58.827 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509455 -lt 64424509455 2023-01-24T18:33:58.827 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:33:58.828 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345933 2023-01-24T18:33:58.828 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:33:58.830 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:33:58.831 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345933 2023-01-24T18:33:58.831 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:33:58.833 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345933 2023-01-24T18:33:58.833 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2023-01-24T18:33:58.833 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345933 2023-01-24T18:33:58.834 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:33:59.341 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345934 -lt 85899345933 2023-01-24T18:33:59.341 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:33:59.342 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182410 2023-01-24T18:33:59.343 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:33:59.344 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:33:59.345 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182410 2023-01-24T18:33:59.346 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:33:59.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182410 2023-01-24T18:33:59.348 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2023-01-24T18:33:59.348 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182410 2023-01-24T18:33:59.349 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:33:59.870 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182410 -lt 107374182410 2023-01-24T18:33:59.870 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-24T18:33:59.871 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:33:59.871 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:34:00.412 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 10 == 0 2023-01-24T18:34:00.412 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:34:00.413 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:34:00.413 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:34:00.413 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:34:00.414 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:34:00.414 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:34:00.414 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:34:00.810 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=10 2023-01-24T18:34:00.811 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:34:00.812 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:34:00.812 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:34:01.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 10 = 10 2023-01-24T18:34:01.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-24T18:34:01.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-24T18:34:01.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:74: TEST_backfill_priority: ceph pg dump pgs 2023-01-24T18:34:01.733 INFO:tasks.workunit.client.0.smithi082.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-24T18:34:01.734 INFO:tasks.workunit.client.0.smithi082.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:50.286306+0000 0'0 78:20 [2,1] 2 [2,1] 2 0'0 2023-01-24T18:33:47.179662+0000 0'0 2023-01-24T18:33:47.179662+0000 0 0 periodic scrub scheduled @ 2023-01-26T02:48:03.535274+0000 0 0 2023-01-24T18:34:01.734 INFO:tasks.workunit.client.0.smithi082.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:46.789381+0000 0'0 77:24 [1,2] 1 [1,2] 1 0'0 2023-01-24T18:33:43.939255+0000 0'0 2023-01-24T18:33:43.939255+0000 0 0 periodic scrub scheduled @ 2023-01-25T21:16:57.426755+0000 0 0 2023-01-24T18:34:01.734 INFO:tasks.workunit.client.0.smithi082.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:44.375416+0000 0'0 77:30 [3,4] 3 [3,4] 3 0'0 2023-01-24T18:33:40.884018+0000 0'0 2023-01-24T18:33:40.884018+0000 0 0 periodic scrub scheduled @ 2023-01-26T00:54:12.055967+0000 0 0 2023-01-24T18:34:01.734 INFO:tasks.workunit.client.0.smithi082.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:41.945384+0000 0'0 77:32 [1,4] 1 [1,4] 1 0'0 2023-01-24T18:33:38.234069+0000 0'0 2023-01-24T18:33:38.234069+0000 0 0 periodic scrub scheduled @ 2023-01-25T23:06:44.631939+0000 0 0 2023-01-24T18:34:01.735 INFO:tasks.workunit.client.0.smithi082.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:39.758234+0000 0'0 77:34 [4,2] 4 [4,2] 4 0'0 2023-01-24T18:33:35.638024+0000 0'0 2023-01-24T18:33:35.638024+0000 0 0 periodic scrub scheduled @ 2023-01-26T05:26:35.498834+0000 0 0 2023-01-24T18:34:01.735 INFO:tasks.workunit.client.0.smithi082.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:35.668941+0000 0'0 77:41 [3,4] 3 [3,4] 3 0'0 2023-01-24T18:33:32.798082+0000 0'0 2023-01-24T18:33:32.798082+0000 0 0 periodic scrub scheduled @ 2023-01-26T02:17:05.107570+0000 0 0 2023-01-24T18:34:01.735 INFO:tasks.workunit.client.0.smithi082.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:32.701617+0000 0'0 77:42 [4,0] 4 [4,0] 4 0'0 2023-01-24T18:33:30.078669+0000 0'0 2023-01-24T18:33:30.078669+0000 0 0 periodic scrub scheduled @ 2023-01-26T00:05:52.762884+0000 0 0 2023-01-24T18:34:01.735 INFO:tasks.workunit.client.0.smithi082.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:30.769483+0000 0'0 77:45 [1,2] 1 [1,2] 1 0'0 2023-01-24T18:33:27.363564+0000 0'0 2023-01-24T18:33:27.363564+0000 0 0 periodic scrub scheduled @ 2023-01-25T23:23:15.447774+0000 0 0 2023-01-24T18:34:01.735 INFO:tasks.workunit.client.0.smithi082.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:28.076599+0000 0'0 77:52 [3,1] 3 [3,1] 3 0'0 2023-01-24T18:33:24.601483+0000 0'0 2023-01-24T18:33:24.601483+0000 0 0 periodic scrub scheduled @ 2023-01-26T01:07:36.844598+0000 0 0 2023-01-24T18:34:01.736 INFO:tasks.workunit.client.0.smithi082.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-24T18:33:24.726045+0000 0'0 77:54 [1,0] 1 [1,0] 1 0'0 2023-01-24T18:33:21.884747+0000 0'0 2023-01-24T18:33:21.884747+0000 0 0 periodic scrub scheduled @ 2023-01-25T21:02:08.951579+0000 0 0 2023-01-24T18:34:01.736 INFO:tasks.workunit.client.0.smithi082.stdout: 2023-01-24T18:34:01.736 INFO:tasks.workunit.client.0.smithi082.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-24T18:34:01.736 INFO:tasks.workunit.client.0.smithi082.stderr:dumped pgs 2023-01-24T18:34:01.745 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:78: TEST_backfill_priority: local PG1 2023-01-24T18:34:01.745 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:79: TEST_backfill_priority: local POOLNUM1 2023-01-24T18:34:01.745 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:80: TEST_backfill_priority: local pool1 2023-01-24T18:34:01.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:81: TEST_backfill_priority: local chk_osd1_1 2023-01-24T18:34:01.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:82: TEST_backfill_priority: local chk_osd1_2 2023-01-24T18:34:01.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:84: TEST_backfill_priority: local PG2 2023-01-24T18:34:01.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:85: TEST_backfill_priority: local POOLNUM2 2023-01-24T18:34:01.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:86: TEST_backfill_priority: local pool2 2023-01-24T18:34:01.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:87: TEST_backfill_priority: local chk_osd2 2023-01-24T18:34:01.747 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:89: TEST_backfill_priority: local PG3 2023-01-24T18:34:01.747 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:90: TEST_backfill_priority: local POOLNUM3 2023-01-24T18:34:01.747 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:91: TEST_backfill_priority: local pool3 2023-01-24T18:34:01.747 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: seq 1 10 2023-01-24T18:34:01.748 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:01.748 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 1.0 --format=json 2023-01-24T18:34:01.748 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:02.172 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:02.173 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2023-01-24T18:34:02.174 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:02.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=0 2023-01-24T18:34:02.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z '' ']' 2023-01-24T18:34:02.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:100: TEST_backfill_priority: PG1=1.0 2023-01-24T18:34:02.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:101: TEST_backfill_priority: POOLNUM1=1 2023-01-24T18:34:02.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:102: TEST_backfill_priority: pool1=test1 2023-01-24T18:34:02.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:103: TEST_backfill_priority: chk_osd1_1=1 2023-01-24T18:34:02.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:104: TEST_backfill_priority: chk_osd1_2=0 2023-01-24T18:34:02.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:02.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 2.0 --format=json 2023-01-24T18:34:02.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:02.595 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:02.597 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=3 2023-01-24T18:34:02.598 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:02.599 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=1 2023-01-24T18:34:02.599 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2023-01-24T18:34:02.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2023-01-24T18:34:02.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2023-01-24T18:34:02.600 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:02.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 3.0 --format=json 2023-01-24T18:34:02.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:03.055 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:03.056 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2023-01-24T18:34:03.057 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:03.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=2 2023-01-24T18:34:03.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2023-01-24T18:34:03.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2023-01-24T18:34:03.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:107: TEST_backfill_priority: PG2=3.0 2023-01-24T18:34:03.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:108: TEST_backfill_priority: POOLNUM2=3 2023-01-24T18:34:03.060 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:109: TEST_backfill_priority: pool2=test3 2023-01-24T18:34:03.060 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:110: TEST_backfill_priority: chk_osd2=2 2023-01-24T18:34:03.060 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:03.060 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 4.0 --format=json 2023-01-24T18:34:03.061 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:03.477 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:03.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=4 2023-01-24T18:34:03.479 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:03.480 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=0 2023-01-24T18:34:03.480 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2023-01-24T18:34:03.480 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2023-01-24T18:34:03.480 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2023-01-24T18:34:03.480 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:03.481 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 5.0 --format=json 2023-01-24T18:34:03.481 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:03.932 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:03.933 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=3 2023-01-24T18:34:03.934 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:03.936 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=4 2023-01-24T18:34:03.936 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2023-01-24T18:34:03.936 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2023-01-24T18:34:03.937 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2023-01-24T18:34:03.937 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:03.937 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 6.0 --format=json 2023-01-24T18:34:03.938 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:04.322 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:04.322 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=4 2023-01-24T18:34:04.323 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:04.325 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=2 2023-01-24T18:34:04.325 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2023-01-24T18:34:04.325 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2023-01-24T18:34:04.326 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2023-01-24T18:34:04.326 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:04.326 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 7.0 --format=json 2023-01-24T18:34:04.327 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2023-01-24T18:34:04.720 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2023-01-24T18:34:04.722 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2023-01-24T18:34:04.723 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2023-01-24T18:34:04.725 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=4 2023-01-24T18:34:04.725 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2023-01-24T18:34:04.725 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2023-01-24T18:34:04.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2023-01-24T18:34:04.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:113: TEST_backfill_priority: PG3=7.0 2023-01-24T18:34:04.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:114: TEST_backfill_priority: POOLNUM3=7 2023-01-24T18:34:04.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:115: TEST_backfill_priority: pool3=test7 2023-01-24T18:34:04.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:116: TEST_backfill_priority: break 2023-01-24T18:34:04.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:119: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2023-01-24T18:34:04.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:121: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2023-01-24T18:34:04.727 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: seq 1 10 2023-01-24T18:34:04.728 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:04.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2023-01-24T18:34:04.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:04.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2023-01-24T18:34:04.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test2 2023-01-24T18:34:04.729 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2023-01-24T18:34:04.730 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2023-01-24T18:34:05.305 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test2' does not exist 2023-01-24T18:34:05.313 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:05.314 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2023-01-24T18:34:05.314 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:05.314 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2023-01-24T18:34:05.314 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test4 2023-01-24T18:34:05.314 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2023-01-24T18:34:05.315 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2023-01-24T18:34:05.926 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test4' does not exist 2023-01-24T18:34:05.935 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:05.935 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2023-01-24T18:34:05.935 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test5 2023-01-24T18:34:05.935 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2023-01-24T18:34:05.936 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2023-01-24T18:34:06.618 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test5' does not exist 2023-01-24T18:34:06.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:06.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2023-01-24T18:34:06.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test6 2023-01-24T18:34:06.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2023-01-24T18:34:06.630 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2023-01-24T18:34:07.466 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test6' does not exist 2023-01-24T18:34:07.477 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:07.477 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2023-01-24T18:34:07.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:07.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2023-01-24T18:34:07.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test8 2023-01-24T18:34:07.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2023-01-24T18:34:07.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2023-01-24T18:34:08.036 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test8' does not exist 2023-01-24T18:34:08.044 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:08.044 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2023-01-24T18:34:08.044 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test9 2023-01-24T18:34:08.045 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2023-01-24T18:34:08.045 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2023-01-24T18:34:08.841 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test9' does not exist 2023-01-24T18:34:08.849 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2023-01-24T18:34:08.850 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2023-01-24T18:34:08.850 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test10 2023-01-24T18:34:08.850 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2023-01-24T18:34:08.850 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2023-01-24T18:34:09.678 INFO:tasks.workunit.client.0.smithi082.stderr:pool 'test10' does not exist 2023-01-24T18:34:09.689 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:135: TEST_backfill_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2023-01-24T18:34:10.531 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 3 size to 1 2023-01-24T18:34:10.543 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:136: TEST_backfill_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2023-01-24T18:34:11.848 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 7 size to 1 2023-01-24T18:34:11.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:137: TEST_backfill_priority: wait_for_clean 2023-01-24T18:34:11.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-24T18:34:11.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-24T18:34:11.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-24T18:34:11.861 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-24T18:34:11.861 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-24T18:34:11.862 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-24T18:34:11.862 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-24T18:34:11.863 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-24T18:34:11.863 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-24T18:34:11.863 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-24T18:34:12.069 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-24T18:34:12.069 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-24T18:34:12.069 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-24T18:34:12.069 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:34:12.070 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:34:12.540 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:34:12.540 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:34:12.541 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:34:12.541 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:34:12.541 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:34:12.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:34:12.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:34:12.542 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:34:12.867 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836505 2023-01-24T18:34:12.867 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836505 2023-01-24T18:34:12.867 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505' 2023-01-24T18:34:12.868 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:34:12.868 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:34:13.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672983 2023-01-24T18:34:13.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672983 2023-01-24T18:34:13.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2023-01-24T18:34:13.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:34:13.018 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:34:13.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509460 2023-01-24T18:34:13.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509460 2023-01-24T18:34:13.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509460' 2023-01-24T18:34:13.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:34:13.180 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:34:13.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345939 2023-01-24T18:34:13.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345939 2023-01-24T18:34:13.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509460 3-85899345939' 2023-01-24T18:34:13.339 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:34:13.339 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:34:13.495 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182415 2023-01-24T18:34:13.496 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182415 2023-01-24T18:34:13.496 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509460 3-85899345939 4-107374182415' 2023-01-24T18:34:13.496 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:34:13.497 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836505 2023-01-24T18:34:13.497 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:34:13.499 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:34:13.500 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836505 2023-01-24T18:34:13.500 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:34:13.501 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836505 2023-01-24T18:34:13.501 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2023-01-24T18:34:13.502 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836505 2023-01-24T18:34:13.502 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:34:13.901 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836505 -lt 21474836505 2023-01-24T18:34:13.901 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:34:13.902 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672983 2023-01-24T18:34:13.902 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:34:13.903 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:34:13.905 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:34:13.905 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672983 2023-01-24T18:34:13.906 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672983 2023-01-24T18:34:13.906 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2023-01-24T18:34:13.907 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949672983 2023-01-24T18:34:13.907 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:34:14.414 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672983 -lt 42949672983 2023-01-24T18:34:14.415 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:34:14.415 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509460 2023-01-24T18:34:14.415 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:34:14.417 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:34:14.418 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509460 2023-01-24T18:34:14.418 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:34:14.420 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509460 2023-01-24T18:34:14.420 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2023-01-24T18:34:14.421 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509460 2023-01-24T18:34:14.421 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:34:14.869 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509458 -lt 64424509460 2023-01-24T18:34:14.870 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:34:15.871 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:34:15.872 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:34:16.313 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509460 -lt 64424509460 2023-01-24T18:34:16.314 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:34:16.315 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345939 2023-01-24T18:34:16.315 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:34:16.316 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:34:16.317 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345939 2023-01-24T18:34:16.317 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:34:16.319 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345939 2023-01-24T18:34:16.320 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2023-01-24T18:34:16.320 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345939 2023-01-24T18:34:16.320 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:34:16.821 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345939 -lt 85899345939 2023-01-24T18:34:16.822 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:34:16.822 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182415 2023-01-24T18:34:16.823 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:34:16.825 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:34:16.826 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182415 2023-01-24T18:34:16.826 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:34:16.828 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182415 2023-01-24T18:34:16.828 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2023-01-24T18:34:16.828 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182415 2023-01-24T18:34:16.829 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:34:17.276 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182416 -lt 107374182415 2023-01-24T18:34:17.277 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-24T18:34:17.277 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:34:17.278 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:34:17.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 3 == 0 2023-01-24T18:34:17.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:34:17.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:34:17.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:34:17.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:34:17.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:34:17.803 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:34:17.803 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:34:18.209 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=3 2023-01-24T18:34:18.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:34:18.210 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:34:18.210 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:34:18.716 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 3 = 3 2023-01-24T18:34:18.716 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-24T18:34:18.716 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-24T18:34:18.716 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:139: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2023-01-24T18:34:18.753 INFO:tasks.workunit.client.0.smithi082.stderr:10+0 records in 2023-01-24T18:34:18.754 INFO:tasks.workunit.client.0.smithi082.stderr:10+0 records out 2023-01-24T18:34:18.754 INFO:tasks.workunit.client.0.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0362916 s, 289 MB/s 2023-01-24T18:34:18.754 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:140: TEST_backfill_priority: p=1 2023-01-24T18:34:18.754 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2023-01-24T18:34:18.755 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2023-01-24T18:34:18.756 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:18.756 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2023-01-24T18:34:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:19.473 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2023-01-24T18:34:20.143 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:20.143 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2023-01-24T18:34:20.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:20.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2023-01-24T18:34:21.732 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:21.732 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2023-01-24T18:34:22.451 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:22.451 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2023-01-24T18:34:23.195 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:23.196 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2023-01-24T18:34:23.915 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:23.916 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2023-01-24T18:34:24.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:24.611 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2023-01-24T18:34:25.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:25.339 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2023-01-24T18:34:26.016 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:26.017 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2023-01-24T18:34:26.719 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:26.719 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2023-01-24T18:34:27.477 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:27.477 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2023-01-24T18:34:28.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:28.175 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2023-01-24T18:34:28.895 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:28.895 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2023-01-24T18:34:29.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:29.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2023-01-24T18:34:30.267 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:30.267 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2023-01-24T18:34:30.928 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:30.929 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2023-01-24T18:34:31.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:31.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2023-01-24T18:34:32.375 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:32.376 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2023-01-24T18:34:33.128 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:33.129 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2023-01-24T18:34:33.890 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:33.890 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2023-01-24T18:34:34.601 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:34.602 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2023-01-24T18:34:35.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:35.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2023-01-24T18:34:36.024 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:36.024 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2023-01-24T18:34:36.852 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:36.852 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2023-01-24T18:34:37.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:37.539 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2023-01-24T18:34:38.275 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:38.275 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2023-01-24T18:34:38.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:38.961 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2023-01-24T18:34:39.714 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:39.714 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2023-01-24T18:34:40.383 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:40.383 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2023-01-24T18:34:41.086 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:41.086 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2023-01-24T18:34:41.806 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:41.806 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2023-01-24T18:34:42.517 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:42.518 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2023-01-24T18:34:43.254 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:43.254 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2023-01-24T18:34:43.965 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:43.965 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2023-01-24T18:34:44.701 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:44.702 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2023-01-24T18:34:45.404 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:45.404 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2023-01-24T18:34:46.123 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:46.124 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2023-01-24T18:34:46.844 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:46.844 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2023-01-24T18:34:47.621 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:47.622 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2023-01-24T18:34:48.358 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:48.359 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2023-01-24T18:34:49.061 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:49.062 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2023-01-24T18:34:49.780 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:49.781 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2023-01-24T18:34:50.501 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:50.501 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2023-01-24T18:34:51.203 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:51.204 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2023-01-24T18:34:51.898 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:51.898 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2023-01-24T18:34:52.818 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:52.818 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2023-01-24T18:34:53.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:53.538 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2023-01-24T18:34:54.257 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:54.257 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2023-01-24T18:34:54.927 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 1 + 1 2023-01-24T18:34:54.929 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=2 2023-01-24T18:34:54.929 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2023-01-24T18:34:54.930 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2023-01-24T18:34:54.931 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:54.932 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2023-01-24T18:34:55.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:55.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2023-01-24T18:34:55.771 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:55.846 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2023-01-24T18:34:56.223 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:56.223 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2023-01-24T18:34:56.607 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:56.608 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2023-01-24T18:34:57.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:57.009 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2023-01-24T18:34:57.477 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:57.478 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2023-01-24T18:34:57.829 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:57.829 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2023-01-24T18:34:58.325 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:58.325 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2023-01-24T18:34:58.682 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:58.682 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2023-01-24T18:34:59.067 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:59.067 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2023-01-24T18:34:59.569 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:59.569 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2023-01-24T18:34:59.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:34:59.962 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2023-01-24T18:35:00.422 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:00.423 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2023-01-24T18:35:00.817 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:00.818 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2023-01-24T18:35:01.232 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:01.233 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2023-01-24T18:35:01.613 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:01.613 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2023-01-24T18:35:01.973 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:01.973 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2023-01-24T18:35:02.416 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:02.416 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2023-01-24T18:35:02.809 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:02.809 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2023-01-24T18:35:03.299 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:03.299 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2023-01-24T18:35:03.687 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:03.688 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2023-01-24T18:35:04.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:04.116 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2023-01-24T18:35:04.516 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:04.516 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2023-01-24T18:35:04.917 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:04.917 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2023-01-24T18:35:05.335 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:05.336 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2023-01-24T18:35:05.753 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:05.754 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2023-01-24T18:35:06.155 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:06.156 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2023-01-24T18:35:06.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:06.541 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2023-01-24T18:35:06.934 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:06.934 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2023-01-24T18:35:07.369 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:07.369 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2023-01-24T18:35:07.812 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:07.812 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2023-01-24T18:35:08.340 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:08.340 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2023-01-24T18:35:08.700 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:08.701 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2023-01-24T18:35:09.089 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:09.090 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2023-01-24T18:35:09.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:09.537 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2023-01-24T18:35:09.922 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:09.922 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2023-01-24T18:35:10.382 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:10.382 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2023-01-24T18:35:10.742 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:10.742 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2023-01-24T18:35:11.095 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:11.096 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2023-01-24T18:35:11.522 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:11.522 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2023-01-24T18:35:11.907 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:11.907 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2023-01-24T18:35:12.333 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:12.334 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2023-01-24T18:35:12.710 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:12.710 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2023-01-24T18:35:13.195 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:13.195 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2023-01-24T18:35:13.621 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:13.622 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2023-01-24T18:35:14.014 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:14.015 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2023-01-24T18:35:14.458 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:14.458 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2023-01-24T18:35:14.826 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:14.827 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2023-01-24T18:35:15.186 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:15.186 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2023-01-24T18:35:15.580 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 2 + 1 2023-01-24T18:35:15.581 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=3 2023-01-24T18:35:15.582 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2023-01-24T18:35:15.582 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2023-01-24T18:35:15.584 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:15.584 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2023-01-24T18:35:15.972 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:15.972 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2023-01-24T18:35:16.407 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:16.408 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2023-01-24T18:35:16.809 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:16.810 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2023-01-24T18:35:17.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:17.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2023-01-24T18:35:17.587 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:17.587 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2023-01-24T18:35:18.030 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:18.030 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2023-01-24T18:35:18.515 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:18.516 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2023-01-24T18:35:18.884 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:18.884 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2023-01-24T18:35:19.360 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:19.361 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2023-01-24T18:35:19.788 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:19.788 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2023-01-24T18:35:20.239 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:20.239 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2023-01-24T18:35:20.674 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:20.674 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2023-01-24T18:35:21.075 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:21.075 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2023-01-24T18:35:21.502 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:21.502 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2023-01-24T18:35:21.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:21.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2023-01-24T18:35:22.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:22.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2023-01-24T18:35:22.736 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:22.737 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2023-01-24T18:35:23.303 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:23.304 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2023-01-24T18:35:23.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:23.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2023-01-24T18:35:24.213 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:24.213 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2023-01-24T18:35:24.613 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:24.614 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2023-01-24T18:35:24.974 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:24.975 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2023-01-24T18:35:25.419 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:25.419 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2023-01-24T18:35:25.805 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:25.805 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2023-01-24T18:35:26.293 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:26.294 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2023-01-24T18:35:26.704 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:26.705 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2023-01-24T18:35:27.111 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:27.112 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2023-01-24T18:35:27.557 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:27.558 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2023-01-24T18:35:27.952 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:27.952 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2023-01-24T18:35:28.414 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:28.414 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2023-01-24T18:35:28.835 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:28.835 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2023-01-24T18:35:29.202 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:29.202 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2023-01-24T18:35:29.563 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:29.564 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2023-01-24T18:35:29.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:29.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2023-01-24T18:35:30.446 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:30.446 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2023-01-24T18:35:30.840 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:30.840 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2023-01-24T18:35:31.268 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:31.269 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2023-01-24T18:35:31.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:31.658 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2023-01-24T18:35:32.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:32.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2023-01-24T18:35:32.466 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:32.466 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2023-01-24T18:35:32.851 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:32.851 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2023-01-24T18:35:33.391 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:33.391 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2023-01-24T18:35:33.799 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:33.799 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2023-01-24T18:35:34.306 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:34.306 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2023-01-24T18:35:34.676 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:34.676 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2023-01-24T18:35:35.129 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:35.129 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2023-01-24T18:35:35.498 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:35.498 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2023-01-24T18:35:35.893 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:35.893 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2023-01-24T18:35:36.356 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2023-01-24T18:35:36.356 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2023-01-24T18:35:36.759 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 3 + 1 2023-01-24T18:35:36.761 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=4 2023-01-24T18:35:36.762 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:150: TEST_backfill_priority: get_not_primary test1 obj1-p1 2023-01-24T18:35:36.762 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1227: get_not_primary: local poolname=test1 2023-01-24T18:35:36.762 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1228: get_not_primary: local objectname=obj1-p1 2023-01-24T18:35:36.763 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: get_primary test1 obj1-p1 2023-01-24T18:35:36.763 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: get_primary: local poolname=test1 2023-01-24T18:35:36.763 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: get_primary: local objectname=obj1-p1 2023-01-24T18:35:36.764 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: ceph --format json osd map test1 obj1-p1 2023-01-24T18:35:36.764 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: get_primary: jq .acting_primary 2023-01-24T18:35:37.180 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local primary=1 2023-01-24T18:35:37.180 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: get_not_primary: ceph --format json osd map test1 obj1-p1 2023-01-24T18:35:37.181 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2023-01-24T18:35:37.620 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:150: TEST_backfill_priority: local otherosd=0 2023-01-24T18:35:37.621 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph pg dump pgs 2023-01-24T18:35:38.026 INFO:tasks.workunit.client.0.smithi082.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-24T18:35:38.026 INFO:tasks.workunit.client.0.smithi082.stdout:7.0 40 0 0 0 0 417333248 0 0 19 19 active+clean 2023-01-24T18:34:12.440241+0000 91'119 91:169 [1] 1 [1] 1 0'0 2023-01-24T18:33:38.234069+0000 0'0 2023-01-24T18:33:38.234069+0000 0 0 periodic scrub scheduled @ 2023-01-25T19:44:42.604966+0000 0 0 2023-01-24T18:35:38.027 INFO:tasks.workunit.client.0.smithi082.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2023-01-24T18:34:11.089291+0000 91'150 91:213 [1] 1 [1] 1 0'0 2023-01-24T18:33:27.363564+0000 0'0 2023-01-24T18:33:27.363564+0000 0 0 periodic scrub scheduled @ 2023-01-26T05:29:45.631123+0000 0 0 2023-01-24T18:35:38.027 INFO:tasks.workunit.client.0.smithi082.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2023-01-24T18:33:24.726045+0000 91'150 91:214 [1,0] 1 [1,0] 1 0'0 2023-01-24T18:33:21.884747+0000 0'0 2023-01-24T18:33:21.884747+0000 0 0 periodic scrub scheduled @ 2023-01-25T21:02:08.951579+0000 0 0 2023-01-24T18:35:38.027 INFO:tasks.workunit.client.0.smithi082.stdout: 2023-01-24T18:35:38.027 INFO:tasks.workunit.client.0.smithi082.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-24T18:35:38.030 INFO:tasks.workunit.client.0.smithi082.stderr:dumped pgs 2023-01-24T18:35:38.038 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:153: TEST_backfill_priority: ERRORS=0 2023-01-24T18:35:38.038 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:155: TEST_backfill_priority: ceph osd set nobackfill 2023-01-24T18:35:38.841 INFO:tasks.workunit.client.0.smithi082.stderr:nobackfill is set 2023-01-24T18:35:38.855 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:156: TEST_backfill_priority: ceph osd set noout 2023-01-24T18:35:39.895 INFO:tasks.workunit.client.0.smithi082.stderr:noout is set 2023-01-24T18:35:39.907 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph osd pool set test7 size 2 2023-01-24T18:35:40.869 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 7 size to 2 2023-01-24T18:35:40.881 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:161: TEST_backfill_priority: sleep 2 2023-01-24T18:35:42.884 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:35:42.884 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:35:42.885 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:35:42.885 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:35:42.886 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:35:42.886 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:35:42.886 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:35:42.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:35:42.887 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:35:42.977 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:35:42.977 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:35:42.978 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:35:42.978 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:35:42.978 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:35:42.978 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:35:42.978 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:35:42.978 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:35:42.979 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:35:42.979 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:35:42.979 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:42.979 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:35:42.979 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:35:42.979 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:35:42.980 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:35:42.980 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:35:42.980 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:35:42.980 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:35:42.980 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:42.981 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:35:42.986 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: seq 1 10 2023-01-24T18:35:42.987 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2023-01-24T18:35:42.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2023-01-24T18:35:42.988 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2023-01-24T18:35:43.398 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:169: TEST_backfill_priority: break 2023-01-24T18:35:43.398 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:177: TEST_backfill_priority: flush_pg_stats 2023-01-24T18:35:43.398 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:35:43.399 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:35:43.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:35:43.859 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:35:43.859 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:35:43.859 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:35:43.859 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:35:43.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:35:43.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:43.861 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:35:44.019 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836525 2023-01-24T18:35:44.019 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836525 2023-01-24T18:35:44.020 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836525' 2023-01-24T18:35:44.020 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:44.020 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:35:44.182 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673003 2023-01-24T18:35:44.183 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673003 2023-01-24T18:35:44.183 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2023-01-24T18:35:44.183 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:44.183 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:35:44.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509480 2023-01-24T18:35:44.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509480 2023-01-24T18:35:44.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509480' 2023-01-24T18:35:44.339 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:44.339 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:35:44.490 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345959 2023-01-24T18:35:44.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345959 2023-01-24T18:35:44.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509480 3-85899345959' 2023-01-24T18:35:44.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:44.492 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:35:44.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182435 2023-01-24T18:35:44.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182435 2023-01-24T18:35:44.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509480 3-85899345959 4-107374182435' 2023-01-24T18:35:44.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:44.641 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836525 2023-01-24T18:35:44.642 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:44.642 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:35:44.644 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836525 2023-01-24T18:35:44.644 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:44.645 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836525 2023-01-24T18:35:44.645 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2023-01-24T18:35:44.646 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836525 2023-01-24T18:35:44.646 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:35:45.064 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836523 -lt 21474836525 2023-01-24T18:35:45.064 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:35:46.065 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:35:46.066 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:35:46.481 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836526 -lt 21474836525 2023-01-24T18:35:46.481 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:46.482 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673003 2023-01-24T18:35:46.482 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:46.484 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:35:46.486 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673003 2023-01-24T18:35:46.486 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:46.488 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673003 2023-01-24T18:35:46.488 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2023-01-24T18:35:46.489 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949673003 2023-01-24T18:35:46.489 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:35:46.974 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673003 -lt 42949673003 2023-01-24T18:35:46.974 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:46.975 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509480 2023-01-24T18:35:46.975 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:46.977 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:35:46.979 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509480 2023-01-24T18:35:46.979 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:46.980 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509480 2023-01-24T18:35:46.981 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509480 2023-01-24T18:35:46.981 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2023-01-24T18:35:46.982 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:35:47.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509481 -lt 64424509480 2023-01-24T18:35:47.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:47.388 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345959 2023-01-24T18:35:47.388 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:47.390 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:35:47.391 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345959 2023-01-24T18:35:47.391 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:47.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345959 2023-01-24T18:35:47.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2023-01-24T18:35:47.394 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345959 2023-01-24T18:35:47.394 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:35:47.832 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345959 -lt 85899345959 2023-01-24T18:35:47.832 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:47.832 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182435 2023-01-24T18:35:47.833 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:47.835 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:35:47.836 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182435 2023-01-24T18:35:47.836 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:47.837 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182435 2023-01-24T18:35:47.838 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182435 2023-01-24T18:35:47.838 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2023-01-24T18:35:47.838 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:35:48.302 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182436 -lt 107374182435 2023-01-24T18:35:48.303 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:35:48.303 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:35:48.303 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:35:48.304 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:35:48.304 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:35:48.304 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:35:48.305 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:35:48.305 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:35:48.305 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:35:48.386 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:35:48.386 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:35:48.387 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:35:48.387 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:35:48.387 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:35:48.387 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:35:48.388 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:35:48.388 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:35:48.388 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:35:48.388 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:35:48.389 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:48.389 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:35:48.389 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:35:48.389 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:35:48.389 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:35:48.390 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:35:48.390 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:35:48.390 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:35:48.390 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:48.390 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:35:48.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph osd out osd.0 2023-01-24T18:35:48.968 INFO:tasks.workunit.client.0.smithi082.stderr:osd.0 is already out. 2023-01-24T18:35:48.975 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:181: TEST_backfill_priority: sleep 2 2023-01-24T18:35:50.977 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:182: TEST_backfill_priority: flush_pg_stats 2023-01-24T18:35:50.978 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:35:50.978 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:35:51.474 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:35:51.475 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:35:51.475 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:35:51.475 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:35:51.475 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:35:51.475 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:35:51.476 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:51.476 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:35:51.715 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836529 2023-01-24T18:35:51.715 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836529 2023-01-24T18:35:51.715 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836529' 2023-01-24T18:35:51.715 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:51.716 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:35:51.876 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673006 2023-01-24T18:35:51.876 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673006 2023-01-24T18:35:51.876 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836529 1-42949673006' 2023-01-24T18:35:51.877 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:51.877 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:35:52.034 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509484 2023-01-24T18:35:52.034 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509484 2023-01-24T18:35:52.034 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484' 2023-01-24T18:35:52.034 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:52.035 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:35:52.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345962 2023-01-24T18:35:52.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345962 2023-01-24T18:35:52.222 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484 3-85899345962' 2023-01-24T18:35:52.222 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:52.222 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:35:52.383 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182439 2023-01-24T18:35:52.383 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182439 2023-01-24T18:35:52.384 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484 3-85899345962 4-107374182439' 2023-01-24T18:35:52.384 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:52.385 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836529 2023-01-24T18:35:52.385 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:52.386 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:35:52.387 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836529 2023-01-24T18:35:52.387 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:52.389 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836529 2023-01-24T18:35:52.389 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2023-01-24T18:35:52.389 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836529 2023-01-24T18:35:52.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:35:52.797 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836527 -lt 21474836529 2023-01-24T18:35:52.798 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:35:53.799 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:35:53.800 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:35:54.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836529 -lt 21474836529 2023-01-24T18:35:54.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:54.222 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673006 2023-01-24T18:35:54.222 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:54.224 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:35:54.225 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673006 2023-01-24T18:35:54.226 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:54.227 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673006 2023-01-24T18:35:54.228 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949673006 2023-01-24T18:35:54.228 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2023-01-24T18:35:54.229 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:35:54.695 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673007 -lt 42949673006 2023-01-24T18:35:54.695 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:54.696 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509484 2023-01-24T18:35:54.696 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:54.698 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:35:54.699 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509484 2023-01-24T18:35:54.699 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:54.701 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509484 2023-01-24T18:35:54.701 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2023-01-24T18:35:54.702 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509484 2023-01-24T18:35:54.702 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:35:55.164 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509484 -lt 64424509484 2023-01-24T18:35:55.165 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:55.165 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345962 2023-01-24T18:35:55.166 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:55.167 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:35:55.168 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345962 2023-01-24T18:35:55.169 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:55.171 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345962 2023-01-24T18:35:55.171 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345962 2023-01-24T18:35:55.172 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345962' 2023-01-24T18:35:55.172 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:35:55.558 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345963 -lt 85899345962 2023-01-24T18:35:55.558 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:35:55.559 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182439 2023-01-24T18:35:55.559 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:35:55.561 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:35:55.562 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182439 2023-01-24T18:35:55.563 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:35:55.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182439 2023-01-24T18:35:55.565 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2023-01-24T18:35:55.565 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182439 2023-01-24T18:35:55.566 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:35:55.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182439 -lt 107374182439 2023-01-24T18:35:55.990 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:35:55.990 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:35:55.990 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:35:55.991 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:35:55.991 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:35:55.991 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:35:55.992 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:35:55.992 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:35:55.992 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:35:56.073 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:35:56.074 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:35:56.074 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:35:56.074 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:35:56.074 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:35:56.074 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:35:56.075 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:35:56.075 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:35:56.075 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:35:56.075 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:35:56.075 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:35:56.076 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:35:56.076 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:56.076 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:35:56.076 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:56.076 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:35:56.077 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:35:56.077 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:35:56.077 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:35:56.077 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:35:56.077 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:35:56.078 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:56.078 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:35:56.078 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:35:56.078 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:35:56.078 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:35:56.079 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:35:56.079 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:35:56.079 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:35:56.079 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:35:56.079 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:35:56.080 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:184: TEST_backfill_priority: ceph pg dump pgs 2023-01-24T18:35:56.484 INFO:tasks.workunit.client.0.smithi082.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-24T18:35:56.484 INFO:tasks.workunit.client.0.smithi082.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2023-01-24T18:35:43.281903+0000 91'150 103:224 [1,4] 1 [1] 1 0'0 2023-01-24T18:33:38.234069+0000 0'0 2023-01-24T18:33:38.234069+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:48:01.377642+0000 0 0 2023-01-24T18:35:56.484 INFO:tasks.workunit.client.0.smithi082.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2023-01-24T18:34:11.089291+0000 91'150 103:225 [1] 1 [1] 1 0'0 2023-01-24T18:33:27.363564+0000 0'0 2023-01-24T18:33:27.363564+0000 0 0 periodic scrub scheduled @ 2023-01-26T05:29:45.631123+0000 0 0 2023-01-24T18:35:56.485 INFO:tasks.workunit.client.0.smithi082.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2023-01-24T18:35:50.711970+0000 91'150 104:236 [1,2] 1 [1,0] 1 0'0 2023-01-24T18:33:21.884747+0000 0'0 2023-01-24T18:33:21.884747+0000 0 0 periodic scrub scheduled @ 2023-01-26T00:18:56.034756+0000 0 0 2023-01-24T18:35:56.486 INFO:tasks.workunit.client.0.smithi082.stdout: 2023-01-24T18:35:56.486 INFO:tasks.workunit.client.0.smithi082.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-24T18:35:56.487 INFO:tasks.workunit.client.0.smithi082.stderr:dumped pgs 2023-01-24T18:35:56.494 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph osd pool set test3 size 2 2023-01-24T18:35:57.210 INFO:tasks.workunit.client.0.smithi082.stderr:set pool 3 size to 2 2023-01-24T18:35:57.228 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:187: TEST_backfill_priority: sleep 2 2023-01-24T18:35:59.225 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:188: TEST_backfill_priority: flush_pg_stats 2023-01-24T18:35:59.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:35:59.226 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:35:59.645 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:35:59.646 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:35:59.646 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:35:59.647 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:35:59.648 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:35:59.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:35:59.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:59.649 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:35:59.832 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836533 2023-01-24T18:35:59.833 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836533 2023-01-24T18:35:59.833 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533' 2023-01-24T18:35:59.833 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:59.834 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:35:59.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673010 2023-01-24T18:35:59.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673010 2023-01-24T18:35:59.989 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673010' 2023-01-24T18:35:59.990 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:35:59.990 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:36:00.142 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509487 2023-01-24T18:36:00.142 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509487 2023-01-24T18:36:00.142 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509487' 2023-01-24T18:36:00.143 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:00.143 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:36:00.295 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345966 2023-01-24T18:36:00.295 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345966 2023-01-24T18:36:00.296 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509487 3-85899345966' 2023-01-24T18:36:00.296 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:00.296 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:36:00.443 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182442 2023-01-24T18:36:00.443 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182442 2023-01-24T18:36:00.443 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509487 3-85899345966 4-107374182442' 2023-01-24T18:36:00.443 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:00.444 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836533 2023-01-24T18:36:00.444 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:00.445 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:36:00.447 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:00.447 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836533 2023-01-24T18:36:00.448 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836533 2023-01-24T18:36:00.448 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836533 2023-01-24T18:36:00.449 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2023-01-24T18:36:00.449 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:36:00.853 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836530 -lt 21474836533 2023-01-24T18:36:00.854 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:36:01.855 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:36:01.856 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:36:02.354 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836533 -lt 21474836533 2023-01-24T18:36:02.354 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:02.354 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673010 2023-01-24T18:36:02.355 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:02.356 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:36:02.357 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673010 2023-01-24T18:36:02.358 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:02.360 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673010 2023-01-24T18:36:02.360 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2023-01-24T18:36:02.360 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949673010 2023-01-24T18:36:02.361 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:36:02.787 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673010 -lt 42949673010 2023-01-24T18:36:02.788 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:02.788 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509487 2023-01-24T18:36:02.789 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:02.790 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:36:02.791 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509487 2023-01-24T18:36:02.792 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:02.794 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509487 2023-01-24T18:36:02.794 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2023-01-24T18:36:02.795 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509487 2023-01-24T18:36:02.795 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:36:03.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509488 -lt 64424509487 2023-01-24T18:36:03.277 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:03.278 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345966 2023-01-24T18:36:03.278 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:03.280 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:36:03.282 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345966 2023-01-24T18:36:03.282 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:03.283 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345966 2023-01-24T18:36:03.283 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345966' 2023-01-24T18:36:03.284 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345966 2023-01-24T18:36:03.284 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:36:03.739 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345966 -lt 85899345966 2023-01-24T18:36:03.740 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:03.740 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182442 2023-01-24T18:36:03.741 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:03.743 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:36:03.744 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182442 2023-01-24T18:36:03.744 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:03.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182442 2023-01-24T18:36:03.746 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2023-01-24T18:36:03.746 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182442 2023-01-24T18:36:03.747 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:36:04.192 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182443 -lt 107374182442 2023-01-24T18:36:04.193 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:36:04.193 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:36:04.194 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:36:04.194 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:36:04.195 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:36:04.195 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:36:04.195 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:36:04.195 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:36:04.196 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:04.287 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:04.288 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:04.289 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:04.289 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:04.289 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:04.289 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:04.289 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:04.290 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:04.290 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:04.290 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:04.290 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:04.290 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:04.290 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:04.291 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:04.291 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:04.291 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:04.291 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:04.291 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:04.292 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:04.292 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:04.292 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:04.292 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:04.292 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:04.293 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:04.293 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:04.293 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:04.293 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:04.293 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:04.293 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:04.294 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:04.294 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:04.294 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:04.294 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:04.294 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:04.295 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:04.295 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:04.295 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:04.295 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:04.295 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:04.295 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:04.296 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:04.296 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:04.296 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:191: TEST_backfill_priority: ceph pg dump pgs 2023-01-24T18:36:04.673 INFO:tasks.workunit.client.0.smithi082.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-24T18:36:04.673 INFO:tasks.workunit.client.0.smithi082.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2023-01-24T18:35:43.281903+0000 91'150 107:227 [1,4] 1 [1] 1 0'0 2023-01-24T18:33:38.234069+0000 0'0 2023-01-24T18:33:38.234069+0000 0 0 periodic scrub scheduled @ 2023-01-26T03:48:01.377642+0000 0 0 2023-01-24T18:36:04.673 INFO:tasks.workunit.client.0.smithi082.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2023-01-24T18:35:59.223164+0000 91'150 108:237 [1,2] 1 [1] 1 0'0 2023-01-24T18:33:27.363564+0000 0'0 2023-01-24T18:33:27.363564+0000 0 0 periodic scrub scheduled @ 2023-01-25T21:40:19.851113+0000 0 0 2023-01-24T18:36:04.674 INFO:tasks.workunit.client.0.smithi082.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2023-01-24T18:35:50.711970+0000 91'150 107:239 [1,2] 1 [1,0] 1 0'0 2023-01-24T18:33:21.884747+0000 0'0 2023-01-24T18:33:21.884747+0000 0 0 periodic scrub scheduled @ 2023-01-26T00:18:56.034756+0000 0 0 2023-01-24T18:36:04.674 INFO:tasks.workunit.client.0.smithi082.stdout: 2023-01-24T18:36:04.674 INFO:tasks.workunit.client.0.smithi082.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-24T18:36:04.676 INFO:tasks.workunit.client.0.smithi082.stderr:dumped pgs 2023-01-24T18:36:04.686 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:04.686 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2023-01-24T18:36:04.726 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: PRIO=110 2023-01-24T18:36:04.727 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:194: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2023-01-24T18:36:04.727 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:04.727 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2023-01-24T18:36:04.771 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: eval 'ITEM="7.0"' 2023-01-24T18:36:04.771 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: ITEM=7.0 2023-01-24T18:36:04.771 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:202: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2023-01-24T18:36:04.772 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:04.772 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2023-01-24T18:36:04.814 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: PRIO=254 2023-01-24T18:36:04.814 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:208: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2023-01-24T18:36:04.815 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: seq 1 10 2023-01-24T18:36:04.816 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2023-01-24T18:36:04.817 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: ceph pg force-backfill 3.0 2023-01-24T18:36:04.817 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2023-01-24T18:36:05.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:219: TEST_backfill_priority: break 2023-01-24T18:36:05.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:227: TEST_backfill_priority: sleep 2 2023-01-24T18:36:07.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:36:07.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:36:07.209 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:36:07.210 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:36:07.210 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:36:07.210 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:36:07.211 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:36:07.211 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:36:07.211 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:07.308 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:229: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:07.309 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:07.310 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:07.310 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:07.310 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:07.310 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:07.310 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:07.311 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:07.311 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:07.311 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:07.311 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:07.311 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:07.312 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:07.312 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:07.312 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:07.312 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:07.312 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:07.312 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 254, 2023-01-24T18:36:07.313 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:07.313 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:07.313 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:07.313 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:07.313 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:07.314 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:07.314 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:07.314 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:07.314 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:07.314 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:07.314 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:07.315 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:07.315 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:07.315 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:07.315 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:07.315 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:07.315 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:07.316 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:07.316 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:07.316 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:07.316 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:07.316 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:07.317 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:07.317 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:07.317 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:07.317 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2023-01-24T18:36:07.356 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: PRIO=254 2023-01-24T18:36:07.357 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:231: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2023-01-24T18:36:07.357 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:236: TEST_backfill_priority: flush_pg_stats 2023-01-24T18:36:07.357 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:36:07.358 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:36:07.823 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:36:07.824 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:36:07.824 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:36:07.824 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:36:07.824 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:36:07.824 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:36:07.825 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:07.825 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:36:08.042 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836536 2023-01-24T18:36:08.042 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836536 2023-01-24T18:36:08.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836536' 2023-01-24T18:36:08.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:08.043 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:36:08.191 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673014 2023-01-24T18:36:08.192 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673014 2023-01-24T18:36:08.192 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836536 1-42949673014' 2023-01-24T18:36:08.192 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:08.193 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:36:08.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509491 2023-01-24T18:36:08.347 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509491 2023-01-24T18:36:08.348 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836536 1-42949673014 2-64424509491' 2023-01-24T18:36:08.348 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:08.348 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:36:08.498 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345970 2023-01-24T18:36:08.498 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345970 2023-01-24T18:36:08.499 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836536 1-42949673014 2-64424509491 3-85899345970' 2023-01-24T18:36:08.499 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:08.499 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:36:08.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182446 2023-01-24T18:36:08.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182446 2023-01-24T18:36:08.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836536 1-42949673014 2-64424509491 3-85899345970 4-107374182446' 2023-01-24T18:36:08.654 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:08.654 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836536 2023-01-24T18:36:08.654 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:08.655 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:36:08.656 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836536 2023-01-24T18:36:08.656 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:08.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836536 2023-01-24T18:36:08.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2023-01-24T18:36:08.658 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836536 2023-01-24T18:36:08.658 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:36:09.154 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836534 -lt 21474836536 2023-01-24T18:36:09.155 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:36:10.156 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:36:10.157 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:36:10.569 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836536 -lt 21474836536 2023-01-24T18:36:10.569 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:10.570 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673014 2023-01-24T18:36:10.570 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:10.572 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:36:10.573 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673014 2023-01-24T18:36:10.574 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:10.576 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673014 2023-01-24T18:36:10.576 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673014' 2023-01-24T18:36:10.577 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949673014 2023-01-24T18:36:10.577 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:36:11.040 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673014 -lt 42949673014 2023-01-24T18:36:11.040 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:11.040 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509491 2023-01-24T18:36:11.041 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:11.043 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:36:11.044 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:11.044 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509491 2023-01-24T18:36:11.045 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509491 2023-01-24T18:36:11.045 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509491 2023-01-24T18:36:11.046 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2023-01-24T18:36:11.046 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:36:11.488 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509492 -lt 64424509491 2023-01-24T18:36:11.488 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:11.489 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345970 2023-01-24T18:36:11.489 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:11.491 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:36:11.492 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345970 2023-01-24T18:36:11.492 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:11.494 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345970 2023-01-24T18:36:11.494 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2023-01-24T18:36:11.495 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345970 2023-01-24T18:36:11.495 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:36:11.934 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345970 -lt 85899345970 2023-01-24T18:36:11.934 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:11.935 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182446 2023-01-24T18:36:11.935 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:11.937 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:36:11.938 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182446 2023-01-24T18:36:11.939 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:11.941 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182446 2023-01-24T18:36:11.941 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2023-01-24T18:36:11.941 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182446 2023-01-24T18:36:11.942 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:36:12.343 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182447 -lt 107374182446 2023-01-24T18:36:12.343 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:239: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2023-01-24T18:36:12.753 INFO:tasks.workunit.client.0.smithi082.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2023-01-24T18:36:12.753 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:36:12.760 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:240: TEST_backfill_priority: sleep 2 2023-01-24T18:36:14.764 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:36:14.764 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:36:14.764 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:36:14.765 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:36:14.765 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:36:14.765 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:36:14.766 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:36:14.766 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:36:14.766 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:14.860 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:242: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:14.861 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:14.861 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:14.862 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:14.862 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:14.862 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:14.862 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:14.862 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:14.863 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:14.863 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:14.863 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:14.863 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:14.863 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:14.863 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:14.864 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:14.864 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:14.864 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:14.864 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:14.864 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:14.865 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:14.865 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:14.865 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:14.865 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:14.865 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:14.866 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:14.866 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:14.866 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:14.866 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:14.866 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:14.867 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:14.867 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:14.867 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:14.867 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:14.867 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:14.867 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:14.868 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:14.868 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:14.868 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:14.868 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:14.868 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:14.869 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:14.869 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:14.869 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:14.870 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2023-01-24T18:36:14.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: PRIO=151 2023-01-24T18:36:14.908 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:244: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2023-01-24T18:36:14.909 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:14.909 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2023-01-24T18:36:14.951 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: eval 'ITEM="3.0"' 2023-01-24T18:36:14.951 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: ITEM=3.0 2023-01-24T18:36:14.951 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:251: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2023-01-24T18:36:14.952 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:14.952 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2023-01-24T18:36:14.996 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: PRIO=254 2023-01-24T18:36:14.996 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:257: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2023-01-24T18:36:14.997 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:264: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2023-01-24T18:36:15.379 INFO:tasks.workunit.client.0.smithi082.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2023-01-24T18:36:15.380 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:36:15.387 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:265: TEST_backfill_priority: sleep 5 2023-01-24T18:36:20.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:36:20.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:36:20.390 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:36:20.391 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:36:20.391 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:36:20.392 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:36:20.392 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:36:20.392 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:36:20.393 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:20.481 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:20.481 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:20.481 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:20.481 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:20.481 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:20.482 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:20.482 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:20.482 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:20.482 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:20.482 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:20.482 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:20.483 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:20.483 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:20.483 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:20.483 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:20.483 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:20.483 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:20.484 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:20.484 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:20.484 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:20.484 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:20.484 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:20.485 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:20.485 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:20.485 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:20.485 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:20.485 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:20.485 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:20.486 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:20.486 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:20.486 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:20.486 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:20.486 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:20.486 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:20.487 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:20.487 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:20.487 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:20.487 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:20.487 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:20.487 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:20.488 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:20.488 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:269: TEST_backfill_priority: flush_pg_stats 2023-01-24T18:36:20.488 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:36:20.489 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:36:20.951 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:36:20.952 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:36:20.952 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:36:20.952 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:36:20.952 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:36:20.952 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:36:20.953 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:20.953 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:36:21.141 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836541 2023-01-24T18:36:21.141 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836541 2023-01-24T18:36:21.141 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836541' 2023-01-24T18:36:21.142 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:21.142 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:36:21.292 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673018 2023-01-24T18:36:21.292 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673018 2023-01-24T18:36:21.292 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836541 1-42949673018' 2023-01-24T18:36:21.292 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:21.293 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:36:21.445 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509496 2023-01-24T18:36:21.445 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509496 2023-01-24T18:36:21.446 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509496' 2023-01-24T18:36:21.446 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:21.448 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:36:21.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345974 2023-01-24T18:36:21.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345974 2023-01-24T18:36:21.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509496 3-85899345974' 2023-01-24T18:36:21.610 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:21.610 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:36:21.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182451 2023-01-24T18:36:21.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182451 2023-01-24T18:36:21.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509496 3-85899345974 4-107374182451' 2023-01-24T18:36:21.770 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:21.770 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836541 2023-01-24T18:36:21.770 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:21.772 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:36:21.773 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836541 2023-01-24T18:36:21.773 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:21.775 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836541 2023-01-24T18:36:21.775 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2023-01-24T18:36:21.776 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836541 2023-01-24T18:36:21.776 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:36:22.211 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836541 -lt 21474836541 2023-01-24T18:36:22.211 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:22.212 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673018 2023-01-24T18:36:22.212 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:22.214 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:36:22.215 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673018 2023-01-24T18:36:22.215 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:22.217 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673018 2023-01-24T18:36:22.218 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673018' 2023-01-24T18:36:22.218 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949673018 2023-01-24T18:36:22.218 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:36:22.621 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673016 -lt 42949673018 2023-01-24T18:36:22.621 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-24T18:36:23.623 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-24T18:36:23.624 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:36:24.038 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673019 -lt 42949673018 2023-01-24T18:36:24.038 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:24.039 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509496 2023-01-24T18:36:24.039 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:24.040 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:36:24.042 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509496 2023-01-24T18:36:24.042 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:24.044 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509496 2023-01-24T18:36:24.045 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2023-01-24T18:36:24.045 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509496 2023-01-24T18:36:24.045 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:36:24.458 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509496 -lt 64424509496 2023-01-24T18:36:24.458 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:24.459 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345974 2023-01-24T18:36:24.459 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:24.461 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:36:24.462 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345974 2023-01-24T18:36:24.462 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:24.464 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345974 2023-01-24T18:36:24.465 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2023-01-24T18:36:24.465 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345974 2023-01-24T18:36:24.465 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:36:24.927 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345974 -lt 85899345974 2023-01-24T18:36:24.927 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:24.928 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182451 2023-01-24T18:36:24.928 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:24.930 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:36:24.931 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182451 2023-01-24T18:36:24.932 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:24.933 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182451 2023-01-24T18:36:24.933 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2023-01-24T18:36:24.934 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182451 2023-01-24T18:36:24.934 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:36:25.356 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182451 -lt 107374182451 2023-01-24T18:36:25.356 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph pg force-backfill 7.0 2023-01-24T18:36:25.759 INFO:tasks.workunit.client.0.smithi082.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2023-01-24T18:36:25.759 INFO:tasks.workunit.client.0.smithi082.stderr: 2023-01-24T18:36:25.774 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:271: TEST_backfill_priority: sleep 2 2023-01-24T18:36:27.777 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:36:27.777 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:36:27.777 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:36:27.778 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:36:27.778 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:36:27.778 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:36:27.779 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:36:27.779 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: CEPH_ARGS= 2023-01-24T18:36:27.779 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:27.876 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:274: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:27.877 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:27.877 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:27.878 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:27.878 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:27.878 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:27.878 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:27.878 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:27.879 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:27.879 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:27.879 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:27.879 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:27.879 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:27.880 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:27.880 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:27.880 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:27.880 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:27.880 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:27.881 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:27.881 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:27.881 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:27.881 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:27.881 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:27.882 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:27.882 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:27.882 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:27.882 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:27.882 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:27.882 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:27.883 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:27.883 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:27.883 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:27.883 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:27.883 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:27.884 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:27.884 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:27.884 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:27.884 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:27.884 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:27.884 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:27.885 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:27.885 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:27.923 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:27.923 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2023-01-24T18:36:27.924 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: PRIO=151 2023-01-24T18:36:27.924 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:276: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2023-01-24T18:36:27.925 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:27.925 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2023-01-24T18:36:27.967 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: eval 'ITEM="7.0"' 2023-01-24T18:36:27.967 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: ITEM=7.0 2023-01-24T18:36:27.968 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:283: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2023-01-24T18:36:27.968 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: cat td/osd-backfill-prio/out 2023-01-24T18:36:27.969 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2023-01-24T18:36:28.011 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: PRIO=254 2023-01-24T18:36:28.012 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:289: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2023-01-24T18:36:28.012 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:296: TEST_backfill_priority: ceph osd unset noout 2023-01-24T18:36:28.780 INFO:tasks.workunit.client.0.smithi082.stderr:noout is unset 2023-01-24T18:36:28.792 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:297: TEST_backfill_priority: ceph osd unset nobackfill 2023-01-24T18:36:29.705 INFO:tasks.workunit.client.0.smithi082.stderr:nobackfill is unset 2023-01-24T18:36:29.718 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:299: TEST_backfill_priority: get_asok_path osd.1 2023-01-24T18:36:29.718 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-01-24T18:36:29.718 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-01-24T18:36:29.719 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-24T18:36:29.719 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-24T18:36:29.720 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.23029 2023-01-24T18:36:29.720 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.23029/ceph-osd.1.asok 2023-01-24T18:36:29.720 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:299: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations' 2023-01-24T18:36:29.721 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations' 2023-01-24T18:36:29.721 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-24T18:36:29.721 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-24T18:36:29.721 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-24T18:36:29.722 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-24T18:36:29.722 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-24T18:36:29.723 INFO:tasks.workunit.client.0.smithi082.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-24T18:36:29.723 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-24T18:36:29.723 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-24T18:36:29.723 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-24T18:36:29.928 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-24T18:36:29.928 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-24T18:36:29.928 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-24T18:36:29.929 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-24T18:36:29.929 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-24T18:36:30.416 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-24T18:36:30.416 INFO:tasks.workunit.client.0.smithi082.stderr:1 2023-01-24T18:36:30.416 INFO:tasks.workunit.client.0.smithi082.stderr:2 2023-01-24T18:36:30.417 INFO:tasks.workunit.client.0.smithi082.stderr:3 2023-01-24T18:36:30.417 INFO:tasks.workunit.client.0.smithi082.stderr:4' 2023-01-24T18:36:30.417 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-24T18:36:30.417 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:30.418 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-24T18:36:30.608 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836545 2023-01-24T18:36:30.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836545 2023-01-24T18:36:30.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836545' 2023-01-24T18:36:30.609 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:30.610 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-24T18:36:30.778 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673022 2023-01-24T18:36:30.778 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673022 2023-01-24T18:36:30.778 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836545 1-42949673022' 2023-01-24T18:36:30.779 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:30.779 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-24T18:36:30.945 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509500 2023-01-24T18:36:30.945 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509500 2023-01-24T18:36:30.945 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836545 1-42949673022 2-64424509500' 2023-01-24T18:36:30.945 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:30.946 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-24T18:36:31.099 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345978 2023-01-24T18:36:31.100 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345978 2023-01-24T18:36:31.100 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836545 1-42949673022 2-64424509500 3-85899345978' 2023-01-24T18:36:31.100 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-24T18:36:31.100 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-24T18:36:31.281 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182455 2023-01-24T18:36:31.282 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182455 2023-01-24T18:36:31.282 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836545 1-42949673022 2-64424509500 3-85899345978 4-107374182455' 2023-01-24T18:36:31.282 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:31.283 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836545 2023-01-24T18:36:31.283 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:31.284 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-24T18:36:31.286 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836545 2023-01-24T18:36:31.286 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:31.287 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836545 2023-01-24T18:36:31.288 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2023-01-24T18:36:31.288 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.0 seq 21474836545 2023-01-24T18:36:31.288 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-24T18:36:31.697 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836545 -lt 21474836545 2023-01-24T18:36:31.697 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:31.698 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673022 2023-01-24T18:36:31.698 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:31.700 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-24T18:36:31.701 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673022 2023-01-24T18:36:31.702 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:31.704 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673022 2023-01-24T18:36:31.704 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2023-01-24T18:36:31.705 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.1 seq 42949673022 2023-01-24T18:36:31.705 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-24T18:36:32.108 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673022 -lt 42949673022 2023-01-24T18:36:32.108 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:32.109 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509500 2023-01-24T18:36:32.109 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:32.111 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-24T18:36:32.112 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509500 2023-01-24T18:36:32.113 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:32.114 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.2 seq 64424509500 2023-01-24T18:36:32.114 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509500 2023-01-24T18:36:32.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2023-01-24T18:36:32.115 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-24T18:36:32.768 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509500 -lt 64424509500 2023-01-24T18:36:32.769 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:32.769 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345978 2023-01-24T18:36:32.770 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:32.771 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-24T18:36:32.772 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345978 2023-01-24T18:36:32.772 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:32.774 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345978 2023-01-24T18:36:32.774 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2023-01-24T18:36:32.774 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.3 seq 85899345978 2023-01-24T18:36:32.775 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-24T18:36:33.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345978 -lt 85899345978 2023-01-24T18:36:33.221 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-24T18:36:33.222 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182455 2023-01-24T18:36:33.222 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-24T18:36:33.223 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-24T18:36:33.224 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182455 2023-01-24T18:36:33.225 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-24T18:36:33.226 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182455 2023-01-24T18:36:33.227 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2023-01-24T18:36:33.227 INFO:tasks.workunit.client.0.smithi082.stdout:waiting osd.4 seq 107374182455 2023-01-24T18:36:33.227 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-24T18:36:33.735 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182455 -lt 107374182455 2023-01-24T18:36:33.736 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-24T18:36:33.737 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:33.737 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:34.485 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 3 == 0 2023-01-24T18:36:34.486 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:34.487 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:34.487 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:34.487 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:34.487 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:34.488 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:34.488 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:34.981 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:34.981 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:34.982 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:34.982 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:35.579 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:35.579 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' -1 2023-01-24T18:36:35.579 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: loop=0 2023-01-24T18:36:35.580 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: num_active_clean=0 2023-01-24T18:36:35.580 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:35.580 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:35.580 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:35.664 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:35.665 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:35.665 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:35.665 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:35.665 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:35.665 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:35.666 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:35.666 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:35.666 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:35.666 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:35.666 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:35.667 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:35.667 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:35.667 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:35.667 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:35.667 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:35.668 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:35.668 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:35.668 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:35.668 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:35.669 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:35.669 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:35.669 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:35.669 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:35.669 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:35.669 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:35.670 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:35.670 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:35.670 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:35.670 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:35.670 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:35.671 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:35.671 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:35.671 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:35.671 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:35.671 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:35.671 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:35.672 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:35.672 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:35.672 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:35.672 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:35.673 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:35.774 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:35.774 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:35.775 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:35.775 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:35.775 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:35.775 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:35.776 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:35.776 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:36.177 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:36.178 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:36.178 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:36.179 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:36.681 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:36.681 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:36.681 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:36.681 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:36.682 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:36.682 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:36.682 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:36.683 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:36.683 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:37.354 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=null 2023-01-24T18:36:37.354 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test null '!=' null 2023-01-24T18:36:37.355 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: (( 1 >= 13 )) 2023-01-24T18:36:37.355 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:37.355 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:37.355 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:37.438 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:37.439 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:37.439 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:37.439 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:37.439 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:37.439 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:37.440 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:37.440 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:37.440 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:37.440 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:37.440 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:37.441 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:37.441 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:37.441 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:37.441 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:37.441 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:37.441 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:37.442 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:37.442 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:37.442 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:37.442 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:37.442 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:37.443 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:37.443 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:37.443 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:37.443 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:37.443 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:37.444 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:37.444 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:37.444 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:37.444 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:37.444 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:37.445 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:37.445 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:37.445 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:37.445 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:37.445 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:37.446 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:37.446 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:37.446 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:37.446 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:37.447 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.2 2023-01-24T18:36:37.647 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:37.648 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:37.649 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:37.649 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:37.649 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:37.649 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:37.650 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:37.650 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:38.089 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:38.090 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:38.090 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:38.091 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:38.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:38.641 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:38.642 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:38.642 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:38.642 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:38.642 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:38.642 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:38.643 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:38.643 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:39.245 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=null 2023-01-24T18:36:39.246 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test null '!=' null 2023-01-24T18:36:39.246 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: (( 2 >= 13 )) 2023-01-24T18:36:39.246 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:39.246 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:39.247 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:39.329 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:39.330 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:39.330 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:39.330 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:39.330 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:39.331 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:39.331 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:39.331 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:39.331 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:39.331 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:39.332 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:39.332 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:39.332 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:39.332 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:39.332 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:39.333 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:39.333 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:39.333 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:39.333 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:39.333 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:39.334 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:39.334 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:39.334 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:39.334 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:39.334 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:39.335 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:39.335 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:39.335 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:39.335 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:39.335 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:39.335 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:39.336 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:39.336 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:39.336 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:39.336 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:39.336 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:39.337 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:39.337 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:39.337 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:39.337 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:39.337 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:39.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.4 2023-01-24T18:36:39.739 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:39.739 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:39.740 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:39.740 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:39.740 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:39.741 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:39.741 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:39.741 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:40.148 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:40.149 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:40.150 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:40.150 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:40.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:40.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:40.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:40.801 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:40.802 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:40.802 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:40.802 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:40.802 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:40.803 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:41.431 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=8235283 2023-01-24T18:36:41.432 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 8235283 '!=' null 2023-01-24T18:36:41.432 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:41.432 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:41.432 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:41.432 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:41.514 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:41.515 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:41.515 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:41.515 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:41.515 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:41.515 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:41.516 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:41.516 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:41.516 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:41.516 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:41.516 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:41.517 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:41.517 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:41.517 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:41.517 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:41.517 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:41.517 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:41.518 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:41.518 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:41.518 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:41.518 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:41.518 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:41.519 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:41.519 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:41.519 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:41.519 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:41.519 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:41.520 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:41.520 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:41.521 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:41.521 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:41.522 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:41.522 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:41.522 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:41.523 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:41.523 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:41.524 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:41.524 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:41.524 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:41.525 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:41.525 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:41.526 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:41.623 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:41.624 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:41.625 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:41.625 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:41.625 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:41.626 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:41.626 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:41.626 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:42.038 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:42.038 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:42.039 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:42.039 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:42.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:42.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:42.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:42.567 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:42.568 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:42.568 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:42.568 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:42.569 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:42.569 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:43.058 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=8235283 2023-01-24T18:36:43.058 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 8235283 '!=' null 2023-01-24T18:36:43.058 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:43.059 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:43.059 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:43.059 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:43.136 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:43.136 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:43.136 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:43.137 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:43.137 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:43.137 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:43.137 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:43.137 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:43.138 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:43.138 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:43.138 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:43.138 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:43.138 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:43.139 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:43.139 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:43.139 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:43.139 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:43.139 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:43.140 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:43.140 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:43.140 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:43.140 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:43.140 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:43.141 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:43.141 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:43.141 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:43.141 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:43.141 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:43.142 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:43.142 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:43.142 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:43.142 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:43.142 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:43.142 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:43.143 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:43.143 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:43.143 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:43.144 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:43.144 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:43.144 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:43.144 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:43.145 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:43.245 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:43.246 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:43.246 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:43.247 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:43.247 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:43.247 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:43.248 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:43.248 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:43.724 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:43.725 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:43.725 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:43.725 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:44.316 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:44.317 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:44.317 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:44.317 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:44.317 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:44.317 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:44.318 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:44.318 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:44.318 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:44.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=15723936 2023-01-24T18:36:44.858 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 15723936 '!=' null 2023-01-24T18:36:44.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:44.859 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:44.859 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:44.859 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:44.942 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:44.943 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:44.943 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:44.943 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:44.943 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:44.944 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:44.944 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:44.944 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:44.944 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:44.944 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:44.945 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:44.945 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:44.945 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:44.945 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:44.945 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:44.945 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:44.946 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:44.946 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:44.946 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:44.946 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:44.946 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:44.946 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:44.947 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:44.947 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:44.947 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:44.947 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:44.948 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:44.948 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:44.948 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:44.948 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:44.948 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:44.948 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:44.949 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:44.949 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:44.949 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:44.949 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:44.949 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:44.950 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:44.950 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:44.950 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:44.950 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:44.951 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:45.051 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:45.052 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:45.052 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:45.053 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:45.053 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:45.053 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:45.054 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:45.054 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:45.454 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:45.454 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:45.455 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:45.455 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:45.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:45.957 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:45.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:45.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:45.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:45.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:45.958 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:45.959 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:45.959 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:46.471 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=15723970 2023-01-24T18:36:46.471 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 15723970 '!=' null 2023-01-24T18:36:46.472 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:46.472 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:46.472 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:46.472 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:46.544 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:46.544 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:46.545 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:46.545 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:46.545 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:46.545 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:46.545 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:46.546 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:46.546 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:46.546 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:46.546 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:46.546 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:46.547 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:46.547 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:46.547 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:46.547 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:46.547 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:46.547 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:46.548 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:46.548 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:46.548 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:46.548 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:46.548 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:46.549 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:46.549 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:46.549 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:46.549 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:46.549 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:46.550 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:46.550 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:46.550 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:46.550 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:46.550 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:46.550 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:46.551 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:46.551 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:46.551 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:46.551 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:46.551 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:46.552 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:46.552 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:46.552 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:46.652 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:46.653 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:46.654 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:46.654 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:46.654 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:46.654 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:46.655 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:46.655 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:47.029 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:47.029 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:47.030 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:47.030 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:47.676 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:47.677 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:47.677 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:47.677 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:47.677 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:47.678 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:47.678 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:47.678 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:47.679 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:48.256 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=15724183 2023-01-24T18:36:48.257 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 15724183 '!=' null 2023-01-24T18:36:48.257 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:48.257 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:48.257 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:48.258 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:48.329 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:48.329 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:48.330 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:48.330 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:48.330 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:48.330 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:48.330 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:48.331 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:48.331 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:48.331 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:48.331 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:48.331 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:48.332 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:48.332 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:48.332 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:48.332 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:48.332 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:48.333 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:48.333 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:48.333 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:48.333 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:48.333 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:48.334 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:48.334 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:48.334 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:48.334 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:48.334 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:48.334 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:48.335 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:48.335 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:48.335 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:48.335 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:48.335 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:48.336 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:48.336 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:48.336 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:48.336 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:48.336 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:48.336 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:48.337 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:48.337 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:48.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:48.438 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:48.438 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:48.439 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:48.439 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:48.439 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:48.440 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:48.440 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:48.440 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:48.841 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:48.841 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:48.842 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:48.842 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:49.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:49.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:49.649 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:49.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:49.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:49.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:49.650 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:49.651 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:49.651 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:50.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=23586326 2023-01-24T18:36:50.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 23586326 '!=' null 2023-01-24T18:36:50.180 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:50.181 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:50.181 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:50.181 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:50.264 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:50.264 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:50.264 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:50.265 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:50.265 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:50.265 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:50.265 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:50.265 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:50.265 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:50.266 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:50.266 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:50.266 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:50.266 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:50.266 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:50.267 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:50.267 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:50.267 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:50.267 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:50.267 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:50.268 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:50.268 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:50.268 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:50.268 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:50.268 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:50.269 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:50.269 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:50.269 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:50.269 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:50.269 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:50.270 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:50.270 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:50.270 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:50.270 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:50.270 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:50.271 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:50.271 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:50.271 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:50.271 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:50.271 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:50.272 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:50.272 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:50.272 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:50.373 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:50.373 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:50.374 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:50.374 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:50.375 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:50.375 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:50.375 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:50.376 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:50.842 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:50.843 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:50.843 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:50.843 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:51.336 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:51.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:51.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:51.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:51.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:51.337 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:51.338 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:51.338 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:51.338 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:52.040 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=15724101 2023-01-24T18:36:52.040 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 15724101 '!=' null 2023-01-24T18:36:52.040 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:52.041 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:52.041 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:52.041 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:52.128 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:52.129 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:52.129 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:52.129 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:52.129 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:52.130 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:52.130 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:52.130 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:52.130 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:52.131 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:52.131 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:52.131 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:52.131 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:52.131 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:52.132 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:52.132 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:52.132 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:52.132 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:52.132 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:52.133 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:52.133 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:52.133 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:52.133 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:52.133 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:52.134 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:52.134 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:52.134 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:52.134 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:52.134 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:52.135 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:52.135 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:52.135 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:52.135 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:52.135 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:52.136 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:52.136 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:52.136 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:52.136 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:52.136 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:52.137 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:52.137 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:52.137 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:52.236 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:52.236 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:52.237 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:52.237 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:52.237 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:52.238 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:52.238 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:52.238 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:52.640 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:52.641 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:52.641 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:52.642 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:53.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:53.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:53.178 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:53.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:53.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:53.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:53.179 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:53.180 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:53.180 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:53.884 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=23585834 2023-01-24T18:36:53.885 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 23585834 '!=' null 2023-01-24T18:36:53.885 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:53.885 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:53.885 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:53.886 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:53.967 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:53.968 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:53.968 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:53.968 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:53.968 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:53.968 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:53.969 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:53.969 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:53.969 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:53.969 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:53.969 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:53.970 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:53.970 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:53.970 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:53.970 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:53.970 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:53.971 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:53.971 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:53.971 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:53.972 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:53.972 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:53.972 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:53.973 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:53.973 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:53.973 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:53.974 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:53.974 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:53.975 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:53.975 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:53.975 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:53.976 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:53.976 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:53.976 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:53.977 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:53.977 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:53.978 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:53.978 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:53.978 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:53.979 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:53.979 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:53.979 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:53.980 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:54.077 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:54.077 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:54.078 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:54.078 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:54.079 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:54.079 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:54.079 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:54.080 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:54.613 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:54.614 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:54.614 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:54.614 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:55.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:55.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:55.205 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:55.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:55.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:55.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:55.206 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:55.207 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:55.207 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:55.841 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=15723750 2023-01-24T18:36:55.841 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 15723750 '!=' null 2023-01-24T18:36:55.841 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:55.842 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:55.842 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: CEPH_ARGS= 2023-01-24T18:36:55.842 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:55.922 INFO:tasks.workunit.client.0.smithi082.stdout:{ 2023-01-24T18:36:55.923 INFO:tasks.workunit.client.0.smithi082.stdout: "local_reservations": { 2023-01-24T18:36:55.923 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:55.923 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:55.923 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [ 2023-01-24T18:36:55.923 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:55.924 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 110, 2023-01-24T18:36:55.924 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:55.924 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:55.924 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "1.0", 2023-01-24T18:36:55.924 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 110, 2023-01-24T18:36:55.925 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:55.925 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:55.925 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:55.925 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:55.926 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:55.926 INFO:tasks.workunit.client.0.smithi082.stdout: "priority": 151, 2023-01-24T18:36:55.926 INFO:tasks.workunit.client.0.smithi082.stdout: "items": [ 2023-01-24T18:36:55.926 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:55.926 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "3.0", 2023-01-24T18:36:55.926 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 151, 2023-01-24T18:36:55.927 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:55.927 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:55.927 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:55.927 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:55.927 INFO:tasks.workunit.client.0.smithi082.stdout: ], 2023-01-24T18:36:55.927 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [ 2023-01-24T18:36:55.928 INFO:tasks.workunit.client.0.smithi082.stdout: { 2023-01-24T18:36:55.928 INFO:tasks.workunit.client.0.smithi082.stdout: "item": "7.0", 2023-01-24T18:36:55.928 INFO:tasks.workunit.client.0.smithi082.stdout: "prio": 254, 2023-01-24T18:36:55.928 INFO:tasks.workunit.client.0.smithi082.stdout: "can_preempt": true 2023-01-24T18:36:55.928 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:55.929 INFO:tasks.workunit.client.0.smithi082.stdout: ] 2023-01-24T18:36:55.929 INFO:tasks.workunit.client.0.smithi082.stdout: }, 2023-01-24T18:36:55.929 INFO:tasks.workunit.client.0.smithi082.stdout: "remote_reservations": { 2023-01-24T18:36:55.929 INFO:tasks.workunit.client.0.smithi082.stdout: "max_allowed": 1, 2023-01-24T18:36:55.929 INFO:tasks.workunit.client.0.smithi082.stdout: "min_priority": 0, 2023-01-24T18:36:55.929 INFO:tasks.workunit.client.0.smithi082.stdout: "queues": [], 2023-01-24T18:36:55.930 INFO:tasks.workunit.client.0.smithi082.stdout: "in_progress": [] 2023-01-24T18:36:55.930 INFO:tasks.workunit.client.0.smithi082.stdout: } 2023-01-24T18:36:55.930 INFO:tasks.workunit.client.0.smithi082.stdout:} 2023-01-24T18:36:55.931 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: sleep 0.1 2023-01-24T18:36:56.031 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: loop+=1 2023-01-24T18:36:56.032 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-24T18:36:56.032 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-24T18:36:56.033 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-24T18:36:56.033 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-24T18:36:56.033 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-24T18:36:56.033 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-24T18:36:56.034 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-24T18:36:56.495 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=0 2023-01-24T18:36:56.496 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-24T18:36:56.496 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-24T18:36:56.497 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-24T18:36:57.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 0 = 3 2023-01-24T18:36:57.115 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: test 0 '!=' 0 2023-01-24T18:36:57.116 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: get_is_making_recovery_progress 2023-01-24T18:36:57.116 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1332: get_is_making_recovery_progress: local recovery_progress 2023-01-24T18:36:57.116 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1333: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-01-24T18:36:57.116 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-01-24T18:36:57.116 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-01-24T18:36:57.117 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: ceph --format json status 2023-01-24T18:36:57.117 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-01-24T18:36:57.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: local progress=15723786 2023-01-24T18:36:57.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: get_is_making_recovery_progress: test 15723786 '!=' null 2023-01-24T18:36:57.657 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: loop=0 2023-01-24T18:36:57.658 INFO:tasks.workunit.client.0.smithi082.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.23029/ceph-osd.1.asok dump_recovery_reservations 2023-01-24T18:36:57.658 INFO:tasks.workunit.client.0.smithi082.stderr://home/ubuntu/cephtest/clone.client.0/q