2022-05-07T15:45:15.019 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-07T15:45:15.020 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} --archive /home/teuthworker/archive/yuriw-2022-05-07_15:11:07-rados-wip-yuri6-testing-2022-05-06-0848-octopus-distro-default-smithi/6825970 --name yuriw-2022-05-07_15:11:07-rados-wip-yuri6-testing-2022-05-06-0848-octopus-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2022-05-07_15:11:07-rados-wip-yuri6-testing-2022-05-06-0848-octopus-distro-default-smithi/6825970/orig.config.yaml 2022-05-07T15:45:15.046 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-05-07T15:45:15.113 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-05-07_15:11:07-rados-wip-yuri6-testing-2022-05-06-0848-octopus-distro-default-smithi/6825970 branch: wip-yuri6-testing-2022-05-06-0848-octopus description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: ceph-qa@ceph.io first_in_suite: false job_id: '6825970' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-05-07_15:11:07-rados-wip-yuri6-testing-2022-05-06-0848-octopus-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri6-testing-2022-05-06-0848-octopus ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 63a8334b02d7dac9af12d1cf5f3ef0666b46826e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 63a8334b02d7dac9af12d1cf5f3ef0666b46826e workunit: branch: wip-yuri6-testing-2022-05-06-0848-octopus sha1: 63a8334b02d7dac9af12d1cf5f3ef0666b46826e owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 63a8334b02d7dac9af12d1cf5f3ef0666b46826e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2022-05-06-0848-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_63a8334b02d7dac9af12d1cf5f3ef0666b46826e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 63a8334b02d7dac9af12d1cf5f3ef0666b46826e targets: smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQjPp3W5Lg2xpJ6PUT4/MwcTtytU3n71xzKYf/cJXnBYnsbGwIVgGp0Tv6WbZu2Zuv6iAqN9LA3wUJJkKGgJyOX37cAQaEOahriIjE6A9aboZwgvJJ2yI7WNmDS2lYArVDowQKTEn/y7t2f6+toC9rwk5E3j+8AX3iqi28qJwOLEycHliVLCXFTP33HA8pCbFl5gOOfV/RoTTB/ekFuIUqJgQHSUMOfFw9JPOgqxoiCyUlhEakVX5avPfUuiEz5VytbrXW7iP5qtOuvshm5057zPgznh8U/GrYKahmEQy/eEJznjoGQFalRVmAdEk6yry5s3HscBsU5MckezBNY5Ep4jmQqZQe8Yqr+CqupQYoAT/RErV6Qj2WJoZlJHn9hgeAcfVuCDtS9sF752bsI1u7/E+OHWpQ09JBErECMlefG7LE2tkLoTL6eEOLQc9kPYfl48A9e5dJ1XzK2MTehaaqpNKYPMA8J2ah/WlUK1ADF4pdXmSWA7LjdBRO2TyJ8NM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master teuthology_sha1: b018fa018d3f885cbc018f839b42536477653f84 timestamp: 2022-05-07_15:11:07 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.456 2022-05-07T15:45:15.113 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_63a8334b02d7dac9af12d1cf5f3ef0666b46826e/qa; will attempt to use it 2022-05-07T15:45:15.114 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_63a8334b02d7dac9af12d1cf5f3ef0666b46826e/qa/tasks 2022-05-07T15:45:15.115 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-05-07T15:45:15.117 INFO:teuthology.task.internal:Checking packages... 2022-05-07T15:45:15.147 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '63a8334b02d7dac9af12d1cf5f3ef0666b46826e' 2022-05-07T15:45:15.148 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-05-07T15:45:15.149 INFO:teuthology.packaging:ref: None 2022-05-07T15:45:15.149 INFO:teuthology.packaging:tag: None 2022-05-07T15:45:15.150 INFO:teuthology.packaging:branch: wip-yuri6-testing-2022-05-06-0848-octopus 2022-05-07T15:45:15.150 INFO:teuthology.packaging:sha1: 63a8334b02d7dac9af12d1cf5f3ef0666b46826e 2022-05-07T15:45:15.151 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri6-testing-2022-05-06-0848-octopus 2022-05-07T15:45:15.318 INFO:teuthology.task.internal:Found packages for ceph version 15.2.16-103-g63a8334b-1focal 2022-05-07T15:45:15.319 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-05-07T15:45:15.329 INFO:teuthology.task.internal:no buildpackages task found 2022-05-07T15:45:15.329 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-05-07T15:45:15.362 INFO:teuthology.task.internal:Saving configuration 2022-05-07T15:45:15.375 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-05-07T15:45:15.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-07T15:45:15.419 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-05-07_15:11:07-rados-wip-yuri6-testing-2022-05-06-0848-octopus-distro-default-smithi/6825970', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-05-07 15:40:19.817661', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBKsNjIaRJvh7AxXPtxHPGm9yWrGM9Wcfote2GCZOOZc'} 2022-05-07T15:45:15.419 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-05-07T15:45:15.431 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2022-05-07T15:45:15.432 INFO:teuthology.run_tasks:Running task console_log... 2022-05-07T15:45:15.495 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9ccee081e0>, signals=[15]) 2022-05-07T15:45:15.497 INFO:teuthology.run_tasks:Running task internal.connect... 2022-05-07T15:45:15.508 INFO:teuthology.task.internal:Opening connections... 2022-05-07T15:45:15.508 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2022-05-07T15:45:15.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-07T15:45:15.648 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-05-07T15:45:15.660 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2022-05-07T15:45:15.769 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2022-05-07T15:45:15.769 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2022-05-07T15:45:15.776 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Ubuntu" 2022-05-07T15:45:15.777 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-05-07T15:45:15.777 INFO:teuthology.orchestra.run.smithi059.stdout:ID=ubuntu 2022-05-07T15:45:15.777 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE=debian 2022-05-07T15:45:15.777 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-05-07T15:45:15.778 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="20.04" 2022-05-07T15:45:15.778 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.ubuntu.com/" 2022-05-07T15:45:15.778 INFO:teuthology.orchestra.run.smithi059.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-05-07T15:45:15.778 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-05-07T15:45:15.778 INFO:teuthology.orchestra.run.smithi059.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-05-07T15:45:15.779 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_CODENAME=focal 2022-05-07T15:45:15.779 INFO:teuthology.orchestra.run.smithi059.stdout:UBUNTU_CODENAME=focal 2022-05-07T15:45:15.780 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2022-05-07T15:45:15.800 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-05-07T15:45:15.822 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-05-07T15:45:15.854 INFO:teuthology.task.internal:Checking for old test directory... 2022-05-07T15:45:15.855 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2022-05-07T15:45:15.861 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-05-07T15:45:15.887 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-05-07T15:45:15.887 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2022-05-07T15:45:16.013 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-05-07T15:45:16.051 INFO:teuthology.run_tasks:Running task kernel... 2022-05-07T15:45:16.070 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-05-07T15:45:16.071 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-05-07T15:45:16.071 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2022-05-07T15:45:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-07T15:45:16.078 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2022-05-07T15:45:16.126 INFO:teuthology.orchestra.run.smithi059.stdout:5.4.0-109-generic 2022-05-07T15:45:16.127 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.4.0-109-generic 2022-05-07T15:45:16.127 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2022-05-07T15:45:16.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2022-05-07T15:45:16.440 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-05-07T15:45:16.452 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-05-07T15:45:16.492 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-05-07T15:45:16.579 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-05-07T15:45:16.807 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [637 kB] 2022-05-07T15:45:16.917 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,750 kB] 2022-05-07T15:45:16.979 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [326 kB] 2022-05-07T15:45:16.982 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.0 kB] 2022-05-07T15:45:16.983 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [920 kB] 2022-05-07T15:45:17.004 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [679 kB] 2022-05-07T15:45:17.015 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [206 kB] 2022-05-07T15:45:17.018 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.7 kB] 2022-05-07T15:45:17.125 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,422 kB] 2022-05-07T15:45:17.526 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [428 kB] 2022-05-07T15:45:17.542 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [246 kB] 2022-05-07T15:45:17.550 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB] 2022-05-07T15:45:17.551 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [702 kB] 2022-05-07T15:45:17.578 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [550 kB] 2022-05-07T15:45:17.621 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [125 kB] 2022-05-07T15:45:17.626 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB] 2022-05-07T15:45:20.282 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 8,388 kB in 2s (5,358 kB/s) 2022-05-07T15:45:21.593 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-05-07T15:45:21.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-05-07T15:45:21.684 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-05-07T15:45:21.846 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-05-07T15:45:21.847 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-05-07T15:45:22.049 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic is already the newest version (5.4.0.109.113). 2022-05-07T15:45:22.049 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic set to manually installed. 2022-05-07T15:45:22.049 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded. 2022-05-07T15:45:22.051 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2022-05-07T15:45:22.072 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2022-05-07T15:45:22.073 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2022-05-07T15:45:22.073 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2022-05-07T15:45:22.073 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2022-05-07T15:45:22.073 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 19 2022-05-07T15:45:22.074 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2022-05-07T15:45:22.074 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2022-05-07T15:45:22.074 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2022-05-07T15:45:22.074 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.4.0.109.113 2022-05-07T15:45:22.075 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.13) 2022-05-07T15:45:22.075 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.4.0-109-generic, linux-modules-extra-5.4.0-109-generic, linux-firmware, intel-microcode, amd64-microcode 2022-05-07T15:45:22.075 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2022-05-07T15:45:22.075 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2022-05-07T15:45:22.076 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2022-05-07T15:45:22.076 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2022-05-07T15:45:22.076 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-05-07T15:45:22.076 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-05-07T15:45:22.077 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-05-07T15:45:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2022-05-07T15:45:22.127 DEBUG:teuthology.parallel:result is None 2022-05-07T15:45:22.127 INFO:teuthology.run_tasks:Running task internal.base... 2022-05-07T15:45:22.138 INFO:teuthology.task.internal:Creating test directory... 2022-05-07T15:45:22.139 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-05-07T15:45:22.146 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-05-07T15:45:22.179 INFO:teuthology.run_tasks:Running task internal.archive... 2022-05-07T15:45:22.213 INFO:teuthology.task.internal:Creating archive directory... 2022-05-07T15:45:22.214 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-05-07T15:45:22.224 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-05-07T15:45:22.249 INFO:teuthology.task.internal:Enabling coredump saving... 2022-05-07T15:45:22.250 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-05-07T15:45:22.287 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:45:22.298 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:45:22.299 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-05-07T15:45:22.322 INFO:teuthology.task.internal:Configuring sudo... 2022-05-07T15:45:22.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-05-07T15:45:22.339 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-05-07T15:45:22.358 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-05-07T15:45:22.358 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-05-07T15:45:22.387 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-07T15:45:22.397 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-07T15:45:22.405 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-05-07T15:45:22.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-05-07T15:45:22.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2022-05-07T15:45:22.523 INFO:teuthology.run_tasks:Running task internal.timer... 2022-05-07T15:45:22.534 INFO:teuthology.task.internal:Starting timer... 2022-05-07T15:45:22.534 INFO:teuthology.run_tasks:Running task pcp... 2022-05-07T15:45:22.571 INFO:teuthology.run_tasks:Running task selinux... 2022-05-07T15:45:22.607 DEBUG:teuthology.task.selinux:Excluding smithi059: OS 'ubuntu' does not support SELinux 2022-05-07T15:45:22.608 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-05-07T15:45:22.608 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-05-07T15:45:22.608 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-05-07T15:45:22.609 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-05-07T15:45:22.640 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-05-07T15:45:22.645 INFO:teuthology.repo_utils:Fetching master from origin 2022-05-07T15:45:22.678 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-05-07T15:45:22.697 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-05-07T15:45:22.699 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-05-07T15:47:34.210 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2022-05-07T15:47:34.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-05-07T15:47:34.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-07T15:47:34.309 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-05-07T15:47:34.436 INFO:teuthology.run_tasks:Running task clock... 2022-05-07T15:47:34.450 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-05-07T15:47:34.451 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-05-07T15:47:34.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-05-07T15:47:34.489 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-05-07T15:47:34.489 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Command line: ntpd -gq 2022-05-07T15:47:34.490 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: proto: precision = 0.058 usec (-24) 2022-05-07T15:47:34.491 INFO:teuthology.orchestra.run.smithi059.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-05-07T15:47:34.491 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-05-07T15:47:34.491 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: restrict ::: KOD does nothing without LIMITED. 2022-05-07T15:47:34.492 INFO:teuthology.orchestra.run.smithi059.stderr:restrict ::: KOD does nothing without LIMITED. 2022-05-07T15:47:34.493 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listen and drop on 0 v6wildcard [::]:123 2022-05-07T15:47:34.493 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-05-07T15:47:34.494 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listen normally on 2 lo 127.0.0.1:123 2022-05-07T15:47:34.494 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listen normally on 3 ens1f0 172.21.15.59:123 2022-05-07T15:47:34.494 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listen normally on 4 lo [::1]:123 2022-05-07T15:47:34.494 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:192a%4]:123 2022-05-07T15:47:34.495 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:34 ntpd[8720]: Listening on routing socket on fd #22 for interface updates 2022-05-07T15:47:35.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:35 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:35.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:35 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:35.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:35 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:35.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:35 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:35.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:35 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:35.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:35 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:36.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:36 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:36.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:36 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:36.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:36 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:36.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:36 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:36.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:36 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:36.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:36 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:37.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:37.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:37.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:37.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:37.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:37.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:37.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:37.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:37.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:37.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:37.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:37.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:37.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:37.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:37.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:37.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:37.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:37.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:37 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:38.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:38 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:38.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:38 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:38.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:38 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:38.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:38 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:38.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:38 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:38.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:38 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:39.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:39.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:39.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:39.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:39.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:39.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:39.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:39.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:39.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:39.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:39.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:39.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:39.496 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:39.496 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:39.496 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:39.496 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:39.497 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:39.497 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:39 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:40.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:40 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:40.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:40 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:40.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:40 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:40.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:40 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:40.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:40 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:40.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:40 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:41.490 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:41.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:41.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:41.491 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:41.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:41.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:41.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:41.492 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:41.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:41.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:41.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-05-07T15:47:41.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/rawstats.20220507: Permission denied 2022-05-07T15:47:41.493 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-05-07T15:47:41.494 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/peerstats.20220507: Permission denied 2022-05-07T15:47:41.494 INFO:teuthology.orchestra.run.smithi059.stdout: 7 May 15:47:41 ntpd[8720]: ntpd: time set -0.179571 s 2022-05-07T15:47:41.494 INFO:teuthology.orchestra.run.smithi059.stdout:ntpd: time set -0.179571s 2022-05-07T15:47:41.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-05-07T15:47:41.495 INFO:teuthology.orchestra.run.smithi059.stderr: 7 May 15:47:41 ntpd[8720]: can't open /var/log/ntpstats/loopstats.20220507: Permission denied 2022-05-07T15:47:41.559 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2022-05-07T15:47:41.559 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2022-05-07T15:47:41.560 INFO:teuthology.orchestra.run.smithi059.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-05-07T15:47:41.560 INFO:teuthology.orchestra.run.smithi059.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-05-07T15:47:41.560 INFO:teuthology.orchestra.run.smithi059.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-05-07T15:47:41.560 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-05-07T15:47:41.561 INFO:teuthology.run_tasks:Running task install... 2022-05-07T15:47:41.573 DEBUG:teuthology.task.install:project ceph 2022-05-07T15:47:41.574 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '63a8334b02d7dac9af12d1cf5f3ef0666b46826e'}} 2022-05-07T15:47:41.574 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '63a8334b02d7dac9af12d1cf5f3ef0666b46826e'} 2022-05-07T15:47:41.575 INFO:teuthology.task.install:Using flavor: default 2022-05-07T15:47:41.581 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-05-07T15:47:41.582 INFO:teuthology.task.install:extra packages: [] 2022-05-07T15:47:41.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-05-07T15:47:41.635 INFO:teuthology.orchestra.run.smithi059.stdout:Ubuntu 2022-05-07T15:47:41.640 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-05-07T15:47:41.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-key list | grep Ceph 2022-05-07T15:47:41.783 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-05-07T15:47:41.885 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-05-07T15:47:41.885 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph.com (release key) 2022-05-07T15:47:41.886 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-05-07T15:47:41.935 INFO:teuthology.orchestra.run.smithi059.stdout:Ubuntu 2022-05-07T15:47:41.940 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-05-07T15:47:41.941 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-05-07T15:47:41.941 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=63a8334b02d7dac9af12d1cf5f3ef0666b46826e 2022-05-07T15:47:42.143 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default/ 2022-05-07T15:47:42.144 INFO:teuthology.task.install.deb:Package version is 15.2.16-103-g63a8334b-1focal 2022-05-07T15:47:42.310 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-05-07T15:47:42.311 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-05-07T15:47:42.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2022-05-07T15:47:42.557 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-05-07T15:47:42.587 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-05-07T15:47:42.625 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-05-07T15:47:42.646 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal InRelease 2022-05-07T15:47:42.666 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-05-07T15:47:42.669 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-05-07T15:47:42.698 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal Release.gpg 2022-05-07T15:47:42.724 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-05-07T15:47:42.763 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main i386 Packages [3,074 B] 2022-05-07T15:47:43.220 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 140 kB in 1s (206 kB/s) 2022-05-07T15:47:44.096 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-05-07T15:47:44.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-103-g63a8334b-1focal cephadm=15.2.16-103-g63a8334b-1focal ceph-mds=15.2.16-103-g63a8334b-1focal ceph-mgr=15.2.16-103-g63a8334b-1focal ceph-common=15.2.16-103-g63a8334b-1focal ceph-fuse=15.2.16-103-g63a8334b-1focal ceph-test=15.2.16-103-g63a8334b-1focal radosgw=15.2.16-103-g63a8334b-1focal python3-rados=15.2.16-103-g63a8334b-1focal python3-rgw=15.2.16-103-g63a8334b-1focal python3-cephfs=15.2.16-103-g63a8334b-1focal python3-rbd=15.2.16-103-g63a8334b-1focal libcephfs2=15.2.16-103-g63a8334b-1focal libcephfs-dev=15.2.16-103-g63a8334b-1focal librados2=15.2.16-103-g63a8334b-1focal librbd1=15.2.16-103-g63a8334b-1focal rbd-fuse=15.2.16-103-g63a8334b-1focal 2022-05-07T15:47:44.188 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-05-07T15:47:44.347 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-05-07T15:47:44.348 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-05-07T15:47:44.489 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2022-05-07T15:47:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-05-07T15:47:44.496 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2022-05-07T15:47:44.496 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2022-05-07T15:47:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-05-07T15:47:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-05-07T15:47:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-05-07T15:47:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-05-07T15:47:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-05-07T15:47:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-05-07T15:47:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-05-07T15:47:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-05-07T15:47:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2022-05-07T15:47:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2022-05-07T15:47:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-05-07T15:47:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-05-07T15:47:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-05-07T15:47:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-05-07T15:47:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-05-07T15:47:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-05-07T15:47:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-05-07T15:47:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-05-07T15:47:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2022-05-07T15:47:44.502 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2022-05-07T15:47:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-05-07T15:47:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-05-07T15:47:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-05-07T15:47:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-05-07T15:47:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-05-07T15:47:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-05-07T15:47:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-05-07T15:47:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-05-07T15:47:44.504 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2022-05-07T15:47:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: btrfs-tools 2022-05-07T15:47:44.596 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-05-07T15:47:44.597 INFO:teuthology.orchestra.run.smithi059.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-05-07T15:47:44.597 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-05-07T15:47:44.597 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-05-07T15:47:44.597 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev libcephfs2 libjq1 2022-05-07T15:47:44.598 INFO:teuthology.orchestra.run.smithi059.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d liblttng-ust0 2022-05-07T15:47:44.598 INFO:teuthology.orchestra.run.smithi059.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-05-07T15:47:44.600 INFO:teuthology.orchestra.run.smithi059.stdout: libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites python3-attr 2022-05-07T15:47:44.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-05-07T15:47:44.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-05-07T15:47:44.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2022-05-07T15:47:44.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2022-05-07T15:47:44.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-05-07T15:47:44.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-05-07T15:47:44.602 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-05-07T15:47:44.602 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-05-07T15:47:44.602 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-05-07T15:47:44.604 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-05-07T15:47:44.604 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-05-07T15:47:44.605 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-05-07T15:47:44.605 INFO:teuthology.orchestra.run.smithi059.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-05-07T15:47:44.605 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-05-07T15:47:44.605 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-05-07T15:47:44.605 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages will be upgraded: 2022-05-07T15:47:44.606 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 librbd1 2022-05-07T15:47:44.755 INFO:teuthology.orchestra.run.smithi059.stdout:2 upgraded, 97 newly installed, 0 to remove and 7 not upgraded. 2022-05-07T15:47:44.755 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 153 MB of archives. 2022-05-07T15:47:44.756 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 697 MB of additional disk space will be used. 2022-05-07T15:47:44.756 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-103-g63a8334b-1focal [1,695 kB] 2022-05-07T15:47:44.785 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-05-07T15:47:44.913 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-103-g63a8334b-1focal [3,248 kB] 2022-05-07T15:47:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-103-g63a8334b-1focal [463 kB] 2022-05-07T15:47:45.028 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-103-g63a8334b-1focal [365 kB] 2022-05-07T15:47:45.044 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-103-g63a8334b-1focal [20.9 kB] 2022-05-07T15:47:45.045 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-103-g63a8334b-1focal [156 kB] 2022-05-07T15:47:45.047 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-103-g63a8334b-1focal [29.4 kB] 2022-05-07T15:47:45.048 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-103-g63a8334b-1focal [363 kB] 2022-05-07T15:47:45.058 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-103-g63a8334b-1focal [3,148 kB] 2022-05-07T15:47:45.163 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-103-g63a8334b-1focal [115 kB] 2022-05-07T15:47:45.166 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-103-g63a8334b-1focal [361 kB] 2022-05-07T15:47:45.173 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-05-07T15:47:45.177 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-103-g63a8334b-1focal [20.1 MB] 2022-05-07T15:47:45.180 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-05-07T15:47:45.193 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-05-07T15:47:45.395 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-05-07T15:47:45.398 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-05-07T15:47:45.496 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-05-07T15:47:45.499 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-05-07T15:47:45.501 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-05-07T15:47:45.502 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-05-07T15:47:45.583 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-05-07T15:47:45.584 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-05-07T15:47:45.585 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-05-07T15:47:45.586 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-05-07T15:47:45.588 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-05-07T15:47:45.639 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-05-07T15:47:45.639 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-05-07T15:47:45.646 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-05-07T15:47:45.649 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-05-07T15:47:45.718 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-05-07T15:47:45.725 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-05-07T15:47:45.726 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-05-07T15:47:45.768 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-05-07T15:47:45.769 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-05-07T15:47:45.799 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-05-07T15:47:45.801 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-05-07T15:47:45.809 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-05-07T15:47:45.914 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-05-07T15:47:45.915 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-103-g63a8334b-1focal [5,342 kB] 2022-05-07T15:47:45.917 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-05-07T15:47:45.917 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-05-07T15:47:45.920 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-05-07T15:47:45.922 INFO:teuthology.orchestra.run.smithi059.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-05-07T15:47:45.973 INFO:teuthology.orchestra.run.smithi059.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-05-07T15:47:45.974 INFO:teuthology.orchestra.run.smithi059.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-05-07T15:47:45.975 INFO:teuthology.orchestra.run.smithi059.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-05-07T15:47:46.025 INFO:teuthology.orchestra.run.smithi059.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-103-g63a8334b-1focal [163 kB] 2022-05-07T15:47:46.029 INFO:teuthology.orchestra.run.smithi059.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-103-g63a8334b-1focal [1,404 kB] 2022-05-07T15:47:46.075 INFO:teuthology.orchestra.run.smithi059.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-103-g63a8334b-1focal [6,170 kB] 2022-05-07T15:47:46.276 INFO:teuthology.orchestra.run.smithi059.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-103-g63a8334b-1focal [24.0 MB] 2022-05-07T15:47:46.796 INFO:teuthology.orchestra.run.smithi059.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-05-07T15:47:46.820 INFO:teuthology.orchestra.run.smithi059.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-05-07T15:47:46.828 INFO:teuthology.orchestra.run.smithi059.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-05-07T15:47:46.984 INFO:teuthology.orchestra.run.smithi059.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-05-07T15:47:46.984 INFO:teuthology.orchestra.run.smithi059.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-05-07T15:47:46.985 INFO:teuthology.orchestra.run.smithi059.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-05-07T15:47:46.985 INFO:teuthology.orchestra.run.smithi059.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-05-07T15:47:46.986 INFO:teuthology.orchestra.run.smithi059.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-05-07T15:47:46.986 INFO:teuthology.orchestra.run.smithi059.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-05-07T15:47:46.987 INFO:teuthology.orchestra.run.smithi059.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-05-07T15:47:46.987 INFO:teuthology.orchestra.run.smithi059.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-05-07T15:47:46.988 INFO:teuthology.orchestra.run.smithi059.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-05-07T15:47:47.065 INFO:teuthology.orchestra.run.smithi059.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-05-07T15:47:47.153 INFO:teuthology.orchestra.run.smithi059.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-05-07T15:47:47.154 INFO:teuthology.orchestra.run.smithi059.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-103-g63a8334b-1focal [4,028 B] 2022-05-07T15:47:47.155 INFO:teuthology.orchestra.run.smithi059.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-103-g63a8334b-1focal [538 kB] 2022-05-07T15:47:47.155 INFO:teuthology.orchestra.run.smithi059.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-05-07T15:47:47.158 INFO:teuthology.orchestra.run.smithi059.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-05-07T15:47:47.205 INFO:teuthology.orchestra.run.smithi059.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-103-g63a8334b-1focal [2,006 kB] 2022-05-07T15:47:47.206 INFO:teuthology.orchestra.run.smithi059.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-05-07T15:47:47.209 INFO:teuthology.orchestra.run.smithi059.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-05-07T15:47:47.211 INFO:teuthology.orchestra.run.smithi059.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-05-07T15:47:47.215 INFO:teuthology.orchestra.run.smithi059.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-05-07T15:47:47.220 INFO:teuthology.orchestra.run.smithi059.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-05-07T15:47:47.220 INFO:teuthology.orchestra.run.smithi059.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-05-07T15:47:47.227 INFO:teuthology.orchestra.run.smithi059.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-103-g63a8334b-1focal [53.2 kB] 2022-05-07T15:47:47.228 INFO:teuthology.orchestra.run.smithi059.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-103-g63a8334b-1focal [58.3 kB] 2022-05-07T15:47:47.228 INFO:teuthology.orchestra.run.smithi059.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-103-g63a8334b-1focal [3,218 kB] 2022-05-07T15:47:47.235 INFO:teuthology.orchestra.run.smithi059.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-05-07T15:47:47.274 INFO:teuthology.orchestra.run.smithi059.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-103-g63a8334b-1focal [34.4 kB] 2022-05-07T15:47:47.275 INFO:teuthology.orchestra.run.smithi059.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-103-g63a8334b-1focal [7,881 kB] 2022-05-07T15:47:47.302 INFO:teuthology.orchestra.run.smithi059.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-05-07T15:47:47.302 INFO:teuthology.orchestra.run.smithi059.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-05-07T15:47:47.401 INFO:teuthology.orchestra.run.smithi059.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-05-07T15:47:47.401 INFO:teuthology.orchestra.run.smithi059.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-05-07T15:47:47.402 INFO:teuthology.orchestra.run.smithi059.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-05-07T15:47:47.402 INFO:teuthology.orchestra.run.smithi059.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-05-07T15:47:47.403 INFO:teuthology.orchestra.run.smithi059.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-05-07T15:47:47.404 INFO:teuthology.orchestra.run.smithi059.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-05-07T15:47:47.404 INFO:teuthology.orchestra.run.smithi059.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-05-07T15:47:47.405 INFO:teuthology.orchestra.run.smithi059.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-05-07T15:47:47.406 INFO:teuthology.orchestra.run.smithi059.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-05-07T15:47:47.541 INFO:teuthology.orchestra.run.smithi059.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-103-g63a8334b-1focal [4,148 B] 2022-05-07T15:47:47.542 INFO:teuthology.orchestra.run.smithi059.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-103-g63a8334b-1focal [39.9 MB] 2022-05-07T15:47:48.975 INFO:teuthology.orchestra.run.smithi059.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-103-g63a8334b-1focal [18.1 kB] 2022-05-07T15:47:48.976 INFO:teuthology.orchestra.run.smithi059.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-103-g63a8334b-1focal [9,665 kB] 2022-05-07T15:47:49.184 INFO:teuthology.orchestra.run.smithi059.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-05-06-0848-octopus/63a8334b02d7dac9af12d1cf5f3ef0666b46826e/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-103-g63a8334b-1focal [60.9 kB] 2022-05-07T15:47:49.815 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 153 MB in 5s (33.1 MB/s) 2022-05-07T15:47:49.979 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-05-07T15:47:50.027 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102177 files and directories currently installed.) 2022-05-07T15:47:50.031 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-05-07T15:47:50.103 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-05-07T15:47:50.527 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../01-librbd1_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:50.656 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librbd1 (15.2.16-103-g63a8334b-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-05-07T15:47:51.207 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../02-librados2_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librados2 (15.2.16-103-g63a8334b-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-05-07T15:47:52.010 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs2. 2022-05-07T15:47:52.025 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../03-libcephfs2_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:52.071 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs2 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:52.390 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rados. 2022-05-07T15:47:52.407 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../04-python3-rados_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:52.449 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rados (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:52.776 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-05-07T15:47:52.792 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../05-python3-ceph-argparse_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:47:52.835 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-argparse (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:53.162 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cephfs. 2022-05-07T15:47:53.175 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../06-python3-cephfs_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:53.212 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cephfs (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:53.532 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-common. 2022-05-07T15:47:53.544 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../07-python3-ceph-common_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:47:53.581 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-common (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:53.901 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-prettytable. 2022-05-07T15:47:53.914 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-05-07T15:47:53.951 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-05-07T15:47:54.228 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rbd. 2022-05-07T15:47:54.243 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../09-python3-rbd_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:54.286 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rbd (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:54.598 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-05-07T15:47:54.611 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../10-librabbitmq4_0.10.0-1_amd64.deb ... 2022-05-07T15:47:54.648 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-05-07T15:47:55.017 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-05-07T15:47:55.031 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../11-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-05-07T15:47:55.075 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-05-07T15:47:55.411 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librgw2. 2022-05-07T15:47:55.426 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../12-librgw2_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:55.461 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librgw2 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:56.038 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rgw. 2022-05-07T15:47:56.051 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../13-python3-rgw_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:56.088 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rgw (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:56.433 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-05-07T15:47:56.445 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../14-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-05-07T15:47:56.485 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-05-07T15:47:56.779 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liboath0. 2022-05-07T15:47:56.795 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-05-07T15:47:56.837 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-05-07T15:47:57.174 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libradosstriper1. 2022-05-07T15:47:57.187 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../16-libradosstriper1_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:57.231 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libradosstriper1 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:57.701 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-common. 2022-05-07T15:47:57.715 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../17-ceph-common_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:57.760 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-common (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:47:59.848 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-base. 2022-05-07T15:47:59.861 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../18-ceph-base_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:47:59.908 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-base (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:00.802 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cryptography. 2022-05-07T15:48:00.815 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-05-07T15:48:00.870 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-05-07T15:48:01.266 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-openssl. 2022-05-07T15:48:01.278 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-05-07T15:48:01.322 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-05-07T15:48:01.591 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-05-07T15:48:01.607 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:48:01.650 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-modules-core (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:02.037 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-bcrypt. 2022-05-07T15:48:02.051 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-05-07T15:48:02.095 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-05-07T15:48:02.381 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cherrypy3. 2022-05-07T15:48:02.394 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-05-07T15:48:02.431 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-05-07T15:48:02.784 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jwt. 2022-05-07T15:48:02.801 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-05-07T15:48:02.926 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-05-07T15:48:03.196 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-logutils. 2022-05-07T15:48:03.210 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-05-07T15:48:03.254 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-05-07T15:48:03.556 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-markupsafe. 2022-05-07T15:48:03.569 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-05-07T15:48:03.608 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-05-07T15:48:03.942 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-mako. 2022-05-07T15:48:03.956 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-05-07T15:48:03.992 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-05-07T15:48:04.329 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplegeneric. 2022-05-07T15:48:04.343 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-05-07T15:48:04.387 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-05-07T15:48:04.665 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-singledispatch. 2022-05-07T15:48:04.679 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-05-07T15:48:04.723 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-05-07T15:48:05.000 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webob. 2022-05-07T15:48:05.016 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-05-07T15:48:05.058 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-05-07T15:48:05.378 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-soupsieve. 2022-05-07T15:48:05.393 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-05-07T15:48:05.436 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-05-07T15:48:05.739 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-bs4. 2022-05-07T15:48:05.751 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-05-07T15:48:05.789 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-05-07T15:48:06.158 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-waitress. 2022-05-07T15:48:06.171 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-05-07T15:48:06.211 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-05-07T15:48:06.586 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempita. 2022-05-07T15:48:06.599 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-05-07T15:48:06.636 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-05-07T15:48:06.939 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-paste. 2022-05-07T15:48:06.951 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-05-07T15:48:07.013 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-05-07T15:48:07.333 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-05-07T15:48:07.346 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-05-07T15:48:07.383 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-05-07T15:48:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastedeploy. 2022-05-07T15:48:07.735 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-05-07T15:48:07.777 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-05-07T15:48:08.080 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webtest. 2022-05-07T15:48:08.096 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-05-07T15:48:08.153 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-05-07T15:48:08.465 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pecan. 2022-05-07T15:48:08.476 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-05-07T15:48:08.513 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-05-07T15:48:08.845 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-werkzeug. 2022-05-07T15:48:08.858 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-05-07T15:48:08.887 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-05-07T15:48:09.265 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr. 2022-05-07T15:48:09.278 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../41-ceph-mgr_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:09.315 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:09.734 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mon. 2022-05-07T15:48:09.749 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../42-ceph-mon_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:09.809 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mon (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:10.796 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-osd. 2022-05-07T15:48:10.810 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../43-ceph-osd_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:10.887 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-osd (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:13.293 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph. 2022-05-07T15:48:13.306 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../44-ceph_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:13.350 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:13.695 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-fuse. 2022-05-07T15:48:13.708 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../45-ceph-fuse_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:13.745 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-fuse (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:14.164 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mds. 2022-05-07T15:48:14.178 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../46-ceph-mds_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:14.222 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mds (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:14.729 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephadm. 2022-05-07T15:48:14.743 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../47-cephadm_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:14.787 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephadm (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:15.098 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jinja2. 2022-05-07T15:48:15.111 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../48-python3-jinja2_2.10.1-2_all.deb ... 2022-05-07T15:48:15.148 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-05-07T15:48:15.492 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-05-07T15:48:15.506 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../49-ceph-mgr-cephadm_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:48:15.551 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-cephadm (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:15.904 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-repoze.lru. 2022-05-07T15:48:15.918 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../50-python3-repoze.lru_0.7-2_all.deb ... 2022-05-07T15:48:15.962 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-05-07T15:48:16.198 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjs-underscore. 2022-05-07T15:48:16.211 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../51-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-05-07T15:48:16.247 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-05-07T15:48:16.534 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-05-07T15:48:16.547 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../52-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-05-07T15:48:16.583 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-05-07T15:48:16.861 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-routes. 2022-05-07T15:48:16.876 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../53-python3-routes_2.4.1-2_all.deb ... 2022-05-07T15:48:16.936 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-05-07T15:48:17.256 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-05-07T15:48:17.271 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../54-ceph-mgr-dashboard_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:48:17.305 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-dashboard (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:18.829 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-05-07T15:48:18.845 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../55-ceph-mgr-diskprediction-cloud_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:48:18.879 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:19.190 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-decorator. 2022-05-07T15:48:19.205 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../56-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-05-07T15:48:19.248 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-05-07T15:48:19.551 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-05-07T15:48:19.564 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../57-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-05-07T15:48:19.601 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-05-07T15:48:19.929 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-scipy. 2022-05-07T15:48:19.943 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../58-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-05-07T15:48:19.987 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-05-07T15:48:21.320 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-05-07T15:48:21.333 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../59-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-05-07T15:48:21.370 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-05-07T15:48:21.915 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-joblib. 2022-05-07T15:48:21.928 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../60-python3-joblib_0.14.0-3_all.deb ... 2022-05-07T15:48:21.965 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-05-07T15:48:22.360 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn. 2022-05-07T15:48:22.374 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../61-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-05-07T15:48:22.418 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-05-07T15:48:23.051 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-05-07T15:48:23.066 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../62-ceph-mgr-diskprediction-local_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:48:23.117 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:24.304 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cachetools. 2022-05-07T15:48:24.319 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../63-python3-cachetools_4.0.0-1_all.deb ... 2022-05-07T15:48:24.362 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-05-07T15:48:24.623 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1. 2022-05-07T15:48:24.636 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../64-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-05-07T15:48:24.673 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-05-07T15:48:24.993 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-05-07T15:48:25.009 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../65-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-05-07T15:48:25.042 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-05-07T15:48:25.295 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rsa. 2022-05-07T15:48:25.310 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../66-python3-rsa_4.0-3_all.deb ... 2022-05-07T15:48:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rsa (4.0-3) ... 2022-05-07T15:48:25.667 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-google-auth. 2022-05-07T15:48:25.683 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../67-python3-google-auth_1.5.1-2_all.deb ... 2022-05-07T15:48:25.725 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-05-07T15:48:26.053 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-blinker. 2022-05-07T15:48:26.069 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../68-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-05-07T15:48:26.111 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-05-07T15:48:26.414 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-oauthlib. 2022-05-07T15:48:26.427 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../69-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-05-07T15:48:26.480 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-05-07T15:48:26.809 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-05-07T15:48:26.822 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../70-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-05-07T15:48:26.858 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-05-07T15:48:27.145 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-websocket. 2022-05-07T15:48:27.161 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../71-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-05-07T15:48:27.195 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-05-07T15:48:27.465 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-kubernetes. 2022-05-07T15:48:27.479 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../72-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-05-07T15:48:27.556 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-05-07T15:48:28.097 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-05-07T15:48:28.111 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../73-ceph-mgr-k8sevents_15.2.16-103-g63a8334b-1focal_all.deb ... 2022-05-07T15:48:28.142 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:28.456 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libonig5:amd64. 2022-05-07T15:48:28.471 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../74-libonig5_6.9.4-1_amd64.deb ... 2022-05-07T15:48:28.515 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-05-07T15:48:28.860 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjq1:amd64. 2022-05-07T15:48:28.874 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../75-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-05-07T15:48:28.918 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-05-07T15:48:29.162 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package jq. 2022-05-07T15:48:29.175 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../76-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-05-07T15:48:29.237 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-05-07T15:48:29.489 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package socat. 2022-05-07T15:48:29.503 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../77-socat_1.7.3.3-2_amd64.deb ... 2022-05-07T15:48:29.548 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking socat (1.7.3.3-2) ... 2022-05-07T15:48:29.976 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-05-07T15:48:29.990 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../78-libxslt1.1_1.1.34-4_amd64.deb ... 2022-05-07T15:48:30.034 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-05-07T15:48:30.304 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package xmlstarlet. 2022-05-07T15:48:30.319 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../79-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-05-07T15:48:30.362 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-05-07T15:48:30.674 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-test. 2022-05-07T15:48:30.687 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../80-ceph-test_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:30.732 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-test (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:34.496 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs-dev. 2022-05-07T15:48:34.512 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../81-libcephfs-dev_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:34.545 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs-dev (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:34.932 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package nvme-cli. 2022-05-07T15:48:34.946 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../82-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-05-07T15:48:35.007 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-05-07T15:48:35.420 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-atomicwrites. 2022-05-07T15:48:35.434 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../83-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-05-07T15:48:35.478 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-05-07T15:48:35.739 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-attr. 2022-05-07T15:48:35.752 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../84-python3-attr_19.3.0-2_all.deb ... 2022-05-07T15:48:35.789 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-05-07T15:48:36.067 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webencodings. 2022-05-07T15:48:36.083 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../85-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-05-07T15:48:36.117 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-05-07T15:48:36.421 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-html5lib. 2022-05-07T15:48:36.434 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../86-python3-html5lib_1.0.1-2_all.deb ... 2022-05-07T15:48:36.477 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-05-07T15:48:36.823 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-05-07T15:48:36.836 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../87-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-05-07T15:48:36.880 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-05-07T15:48:37.376 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-packaging. 2022-05-07T15:48:37.390 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../88-python3-packaging_20.3-1_all.deb ... 2022-05-07T15:48:37.433 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-packaging (20.3-1) ... 2022-05-07T15:48:37.745 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastescript. 2022-05-07T15:48:37.927 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../89-python3-pastescript_2.0.2-4_all.deb ... 2022-05-07T15:48:37.928 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-05-07T15:48:38.139 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pluggy. 2022-05-07T15:48:38.156 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../90-python3-pluggy_0.13.0-2_all.deb ... 2022-05-07T15:48:38.189 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-05-07T15:48:38.509 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-psutil. 2022-05-07T15:48:38.523 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../91-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-05-07T15:48:38.567 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-05-07T15:48:38.903 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-py. 2022-05-07T15:48:38.918 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../92-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-05-07T15:48:38.961 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-05-07T15:48:39.289 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyinotify. 2022-05-07T15:48:39.305 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../93-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-05-07T15:48:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-05-07T15:48:39.659 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-wcwidth. 2022-05-07T15:48:39.674 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../94-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-05-07T15:48:39.717 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-05-07T15:48:40.036 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pytest. 2022-05-07T15:48:40.050 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../95-python3-pytest_4.6.9-1_all.deb ... 2022-05-07T15:48:40.094 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-05-07T15:48:40.456 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplejson. 2022-05-07T15:48:40.471 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../96-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-05-07T15:48:40.514 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-05-07T15:48:40.968 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package radosgw. 2022-05-07T15:48:40.983 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../97-radosgw_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:41.026 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking radosgw (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:42.063 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-fuse. 2022-05-07T15:48:42.077 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../98-rbd-fuse_15.2.16-103-g63a8334b-1focal_amd64.deb ... 2022-05-07T15:48:42.120 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-fuse (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:42.412 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-attr (19.3.0-2) ... 2022-05-07T15:48:42.792 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-05-07T15:48:42.918 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-05-07T15:48:43.069 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-05-07T15:48:43.262 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephadm (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:43.914 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user cephadm....done 2022-05-07T15:48:44.051 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-05-07T15:48:44.427 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-05-07T15:48:44.745 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-05-07T15:48:44.862 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-05-07T15:48:45.181 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-05-07T15:48:45.615 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-05-07T15:48:45.917 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-05-07T15:48:46.201 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-05-07T15:48:46.494 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-05-07T15:48:46.803 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-argparse (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:47.130 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-05-07T15:48:47.238 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-05-07T15:48:47.514 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-05-07T15:48:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-05-07T15:48:48.200 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-05-07T15:48:48.518 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-05-07T15:48:48.869 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-05-07T15:48:49.170 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-05-07T15:48:49.521 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-05-07T15:48:49.822 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-packaging (20.3-1) ... 2022-05-07T15:48:50.107 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-05-07T15:48:50.257 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-05-07T15:48:50.551 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-05-07T15:48:50.969 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-05-07T15:48:51.354 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-05-07T15:48:51.714 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up socat (1.7.3.3-2) ... 2022-05-07T15:48:51.848 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-common (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:52.149 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-05-07T15:48:52.501 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-05-07T15:48:52.626 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-05-07T15:48:52.919 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-05-07T15:48:53.045 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-05-07T15:48:53.380 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-05-07T15:48:53.673 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-05-07T15:48:53.960 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-05-07T15:48:54.083 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-05-07T15:48:54.384 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-05-07T15:48:54.685 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-05-07T15:48:54.986 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-05-07T15:48:55.399 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-05-07T15:48:55.672 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-05-07T15:48:55.781 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-05-07T15:48:56.082 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-05-07T15:48:56.500 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-05-07T15:48:56.626 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-05-07T15:48:56.777 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-05-07T15:48:57.112 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-05-07T15:48:57.404 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librados2 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:57.522 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-05-07T15:48:59.011 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librgw2 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:48:59.136 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-05-07T15:48:59.262 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-05-07T15:48:59.647 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-05-07T15:48:59.907 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-05-07T15:49:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-05-07T15:49:00.559 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs2 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:00.685 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-05-07T15:49:01.032 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libradosstriper1 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:01.171 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-05-07T15:49:01.514 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-05-07T15:49:02.611 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librbd1 (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:02.761 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-05-07T15:49:02.946 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-05-07T15:49:03.072 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-05-07T15:49:03.381 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-05-07T15:49:03.535 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rsa (4.0-3) ... 2022-05-07T15:49:03.801 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-05-07T15:49:04.101 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-modules-core (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:04.227 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-fuse (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:04.471 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-05-07T15:49:04.471 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-05-07T15:49:04.979 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs-dev (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:05.336 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-05-07T15:49:05.847 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-05-07T15:49:06.174 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-05-07T15:49:06.526 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rados (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:06.660 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rbd (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:06.786 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-fuse (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:06.912 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-routes (2.4.1-2) ... 2022-05-07T15:49:07.238 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-05-07T15:49:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rgw (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:07.665 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cephfs (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:08.043 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-05-07T15:49:08.972 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-common (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:09.533 INFO:teuthology.orchestra.run.smithi059.stdout:Adding group ceph....done 2022-05-07T15:49:09.900 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user ceph....done 2022-05-07T15:49:10.120 INFO:teuthology.orchestra.run.smithi059.stdout:Setting system user ceph properties....done 2022-05-07T15:49:10.136 INFO:teuthology.orchestra.run.smithi059.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-05-07T15:49:10.288 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-05-07T15:49:10.633 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-05-07T15:49:11.134 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-test (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:11.318 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up radosgw (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:11.623 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-05-07T15:49:11.623 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-05-07T15:49:12.294 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-base (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:12.679 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-05-07T15:49:13.129 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mds (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:13.351 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-05-07T15:49:13.351 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-05-07T15:49:13.814 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:14.038 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-05-07T15:49:14.039 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-05-07T15:49:14.498 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:14.641 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-osd (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:15.135 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-05-07T15:49:15.135 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-05-07T15:49:15.618 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-k8sevents (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:15.752 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:16.045 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mon (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:16.257 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-05-07T15:49:16.257 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-05-07T15:49:16.722 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-cephadm (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:16.865 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:16.991 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-dashboard (15.2.16-103-g63a8334b-1focal) ... 2022-05-07T15:49:17.160 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for systemd (245.4-4ubuntu3.16) ... 2022-05-07T15:49:17.452 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-05-07T15:49:20.371 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-05-07T15:49:20.807 INFO:teuthology.orchestra.run.smithi059.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-05-07T15:49:20.811 DEBUG:teuthology.parallel:result is None 2022-05-07T15:49:20.812 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-05-07T15:49:20.860 INFO:teuthology.orchestra.run.smithi059.stdout:Ubuntu 2022-05-07T15:49:20.865 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-05-07T15:49:20.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=63a8334b02d7dac9af12d1cf5f3ef0666b46826e 2022-05-07T15:49:21.051 DEBUG:teuthology.orchestra.run.smithi059:> dpkg-query -W -f '${Version}' ceph 2022-05-07T15:49:21.073 INFO:teuthology.orchestra.run.smithi059.stdout:15.2.16-103-g63a8334b-1focal 2022-05-07T15:49:21.074 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-103-g63a8334b-1focal 2022-05-07T15:49:21.075 INFO:teuthology.task.install:The correct ceph version 15.2.16-103-g63a8334b-1focal is installed. 2022-05-07T15:49:21.075 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-05-07T15:49:21.076 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-05-07T15:49:21.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-05-07T15:49:21.088 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-05-07T15:49:21.089 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-05-07T15:49:21.089 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2022-05-07T15:49:21.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-05-07T15:49:21.156 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-05-07T15:49:21.156 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-05-07T15:49:21.157 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2022-05-07T15:49:21.168 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-05-07T15:49:21.227 INFO:teuthology.run_tasks:Running task workunit... 2022-05-07T15:49:21.249 INFO:tasks.workunit:Pulling workunits from ref 63a8334b02d7dac9af12d1cf5f3ef0666b46826e 2022-05-07T15:49:21.251 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-05-07T15:49:21.251 INFO:tasks.workunit:timeout=3h 2022-05-07T15:49:21.251 INFO:tasks.workunit:cleanup=True 2022-05-07T15:49:21.252 DEBUG:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.0 2022-05-07T15:49:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-07T15:49:21.259 INFO:teuthology.orchestra.run.smithi059.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-05-07T15:49:21.260 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-05-07T15:49:21.266 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-05-07T15:49:21.267 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-05-07T15:49:21.273 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 63a8334b02d7dac9af12d1cf5f3ef0666b46826e 2022-05-07T15:49:21.282 INFO:tasks.workunit.client.0.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-05-07T15:50:01.740 INFO:tasks.workunit.client.0.smithi059.stderr:Note: switching to '63a8334b02d7dac9af12d1cf5f3ef0666b46826e'. 2022-05-07T15:50:01.740 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.741 INFO:tasks.workunit.client.0.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-05-07T15:50:01.741 INFO:tasks.workunit.client.0.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2022-05-07T15:50:01.741 INFO:tasks.workunit.client.0.smithi059.stderr:state without impacting any branches by switching back to a branch. 2022-05-07T15:50:01.741 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.742 INFO:tasks.workunit.client.0.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2022-05-07T15:50:01.742 INFO:tasks.workunit.client.0.smithi059.stderr:do so (now or later) by using -c with the switch command. Example: 2022-05-07T15:50:01.742 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.742 INFO:tasks.workunit.client.0.smithi059.stderr: git switch -c 2022-05-07T15:50:01.743 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.743 INFO:tasks.workunit.client.0.smithi059.stderr:Or undo this operation with: 2022-05-07T15:50:01.743 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.743 INFO:tasks.workunit.client.0.smithi059.stderr: git switch - 2022-05-07T15:50:01.744 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.744 INFO:tasks.workunit.client.0.smithi059.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-05-07T15:50:01.744 INFO:tasks.workunit.client.0.smithi059.stderr: 2022-05-07T15:50:01.744 INFO:tasks.workunit.client.0.smithi059.stderr:HEAD is now at 63a8334b02d Merge branch 'backport-45420-octopus' of https://github.com/mgfritch/ceph into wip-yuri6-testing-2022-05-06-0848-octopus 2022-05-07T15:50:01.746 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-05-07T15:50:01.767 DEBUG:teuthology.orchestra.remote:smithi059:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2022-05-07T15:50:01.780 INFO:tasks.workunit:Running workunits matching osd on client.0... 2022-05-07T15:50:01.781 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2022-05-07T15:50:01.781 DEBUG:teuthology.orchestra.run.smithi059:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=63a8334b02d7dac9af12d1cf5f3ef0666b46826e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2022-05-07T15:50:01.797 INFO:tasks.workunit.client.0.smithi059.stderr:stty: 'standard input': Inappropriate ioctl for device 2022-05-07T15:50:01.836 INFO:tasks.workunit.client.0.smithi059.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2022-05-07T15:50:01.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2022-05-07T15:50:01.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2022-05-07T15:50:01.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2022-05-07T15:50:01.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2022-05-07T15:50:01.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2022-05-07T15:50:01.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2022-05-07T15:50:01.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2022-05-07T15:50:01.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2022-05-07T15:50:01.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/bad-inc-map 2022-05-07T15:50:01.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2022-05-07T15:50:01.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2022-05-07T15:50:01.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:45871 2022-05-07T15:50:01.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:45871 2022-05-07T15:50:01.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2022-05-07T15:50:01.840 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2022-05-07T15:50:01.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none ' 2022-05-07T15:50:01.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:45871 ' 2022-05-07T15:50:01.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2022-05-07T15:50:01.841 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2022-05-07T15:50:01.842 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2022-05-07T15:50:01.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2022-05-07T15:50:01.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2022-05-07T15:50:01.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2022-05-07T15:50:01.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2022-05-07T15:50:01.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2022-05-07T15:50:01.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2022-05-07T15:50:01.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:50:01.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2022-05-07T15:50:01.845 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:50:01.845 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:50:01.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:50:01.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:50:01.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:50:01.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:50:01.847 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:50:01.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:50:01.848 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:50:01.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:50:01.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:50:01.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:50:01.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:01.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:50:01.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:50:01.852 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:01.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:50:01.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:50:01.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2022-05-07T15:50:01.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:50:01.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:01.857 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:01.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12171 2022-05-07T15:50:01.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:50:01.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:50:01.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2022-05-07T15:50:01.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-05-07T15:50:01.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:01.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:01.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12171 2022-05-07T15:50:01.862 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-05-07T15:50:01.865 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2022-05-07T15:50:01.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2022-05-07T15:50:01.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-05-07T15:50:01.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2022-05-07T15:50:01.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2022-05-07T15:50:01.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2022-05-07T15:50:01.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2022-05-07T15:50:01.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2022-05-07T15:50:01.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-05-07T15:50:01.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-05-07T15:50:01.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-05-07T15:50:01.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2022-05-07T15:50:01.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2022-05-07T15:50:02.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-05-07T15:50:02.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:02.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:02.316 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:02.316 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:02.316 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:02.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:02.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-05-07T15:50:02.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-05-07T15:50:02.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-05-07T15:50:02.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:50:02.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:50:02.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-05-07T15:50:02.499 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-05-07T15:50:02.499 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:50:02.499 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:50:02.500 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:50:02.501 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:50:02.501 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:02.501 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:02.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12171/ceph-mon.a.asok 2022-05-07T15:50:02.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:50:02.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12171/ceph-mon.a.asok config get fsid 2022-05-07T15:50:02.591 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-05-07T15:50:02.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:50:02.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:50:02.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-05-07T15:50:02.593 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-05-07T15:50:02.593 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:02.595 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:02.595 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12171/ceph-mon.a.asok 2022-05-07T15:50:02.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:50:02.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12171/ceph-mon.a.asok config get mon_host 2022-05-07T15:50:02.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2022-05-07T15:50:02.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/bad-inc-map 2022-05-07T15:50:02.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-05-07T15:50:02.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-05-07T15:50:02.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-05-07T15:50:02.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/bad-inc-map/x 2022-05-07T15:50:02.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-05-07T15:50:03.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-05-07T15:50:03.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:03.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:03.562 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:03.562 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:03.563 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:03.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:03.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:50:03.573 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:50:03.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2022-05-07T15:50:03.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2022-05-07T15:50:03.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:50:03.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-05-07T15:50:03.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:50:03.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/0 2022-05-07T15:50:03.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 ' 2022-05-07T15:50:03.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:50:03.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:50:03.645 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:50:03.645 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2022-05-07T15:50:03.645 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2022-05-07T15:50:03.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:50:03.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:50:03.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2022-05-07T15:50:03.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:50:03.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:03.648 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:03.648 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:03.648 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:03.649 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:03.650 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:03.650 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:50:03.651 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:50:03.651 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:50:03.651 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2022-05-07T15:50:03.652 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2022-05-07T15:50:03.652 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:50:03.653 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:50:03.653 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:03.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:50:03.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:50:03.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/0 2022-05-07T15:50:03.655 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:50:03.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cc7d238a-8931-4342-a38f-286f7e27d4d6 2022-05-07T15:50:03.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 cc7d238a-8931-4342-a38f-286f7e27d4d6' 2022-05-07T15:50:03.657 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 cc7d238a-8931-4342-a38f-286f7e27d4d6 2022-05-07T15:50:03.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:50:03.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQArlXZiA+kaJxAA9NHEGuCbMBlwoa+3dtVVrw== 2022-05-07T15:50:03.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQArlXZiA+kaJxAA9NHEGuCbMBlwoa+3dtVVrw=="}' 2022-05-07T15:50:03.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cc7d238a-8931-4342-a38f-286f7e27d4d6 -i td/bad-inc-map/0/new.json 2022-05-07T15:50:03.967 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:03.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/0/new.json 2022-05-07T15:50:03.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQArlXZiA+kaJxAA9NHEGuCbMBlwoa+3dtVVrw== --osd-uuid cc7d238a-8931-4342-a38f-286f7e27d4d6 2022-05-07T15:50:04.010 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.010+0000 7f0e21f01d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:04.012 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.014+0000 7f0e21f01d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:04.014 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.014+0000 7f0e21f01d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:04.015 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.014+0000 7f0e21f01d80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2022-05-07T15:50:04.015 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.014+0000 7f0e21f01d80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2022-05-07T15:50:04.016 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.014+0000 7f0e21f01d80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2022-05-07T15:50:04.113 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.110+0000 7f0e21f01d80 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2022-05-07T15:50:04.548 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:04.546+0000 7f0e21f01d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:50:07.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/0/keyring 2022-05-07T15:50:07.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:50:07.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-05-07T15:50:07.317 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2022-05-07T15:50:07.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:50:07.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-05-07T15:50:07.564 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2022-05-07T15:50:07.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:07.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:50:07.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:50:07.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:50:07.584 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:07.582+0000 7f7781170d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:07.603 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:07.602+0000 7f7781170d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:07.604 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:07.602+0000 7f7781170d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:07.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-05-07T15:50:07.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:07.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:50:07.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:07.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:07.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:07.765 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:07.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:07.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:07.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:07.910 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:07.910+0000 7f7781170d80 -1 Falling back to public interface 2022-05-07T15:50:07.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:08.757 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:08.758+0000 7f7781170d80 -1 osd.0 0 log_to_monitors {default=true} 2022-05-07T15:50:08.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:08.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:08.955 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:08.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:50:08.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:08.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:09.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:10.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:10.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:10.154 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:50:10.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:50:10.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:10.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:10.585 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/12523,v1:127.0.0.1:6801/12523] [v2:127.0.0.1:6802/12523,v1:127.0.0.1:6803/12523] exists,up cc7d238a-8931-4342-a38f-286f7e27d4d6 2022-05-07T15:50:10.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:10.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:10.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:10.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2022-05-07T15:50:10.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2022-05-07T15:50:10.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:50:10.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-05-07T15:50:10.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:50:10.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/1 2022-05-07T15:50:10.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 ' 2022-05-07T15:50:10.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:50:10.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:50:10.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:50:10.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2022-05-07T15:50:10.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2022-05-07T15:50:10.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:50:10.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:50:10.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2022-05-07T15:50:10.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:50:10.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:10.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:10.594 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:10.594 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:10.595 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:10.595 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:10.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:10.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:50:10.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:50:10.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:50:10.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2022-05-07T15:50:10.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2022-05-07T15:50:10.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:50:10.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:50:10.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:10.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:50:10.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:50:10.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/1 2022-05-07T15:50:10.598 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:50:10.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=77914f20-4f34-4d38-a7bd-57cea0b5dfb9 2022-05-07T15:50:10.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 77914f20-4f34-4d38-a7bd-57cea0b5dfb9' 2022-05-07T15:50:10.599 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 77914f20-4f34-4d38-a7bd-57cea0b5dfb9 2022-05-07T15:50:10.599 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:50:10.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAylXZib0OEJBAAtKAhvYycw6gFVbcJ33I6GA== 2022-05-07T15:50:10.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAylXZib0OEJBAAtKAhvYycw6gFVbcJ33I6GA=="}' 2022-05-07T15:50:10.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 77914f20-4f34-4d38-a7bd-57cea0b5dfb9 -i td/bad-inc-map/1/new.json 2022-05-07T15:50:11.005 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:11.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/1/new.json 2022-05-07T15:50:11.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAylXZib0OEJBAAtKAhvYycw6gFVbcJ33I6GA== --osd-uuid 77914f20-4f34-4d38-a7bd-57cea0b5dfb9 2022-05-07T15:50:11.043 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.042+0000 7fe2c70e7d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:11.046 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.046+0000 7fe2c70e7d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:11.048 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.050+0000 7fe2c70e7d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:11.049 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.050+0000 7fe2c70e7d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2022-05-07T15:50:11.049 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.050+0000 7fe2c70e7d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2022-05-07T15:50:11.050 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.050+0000 7fe2c70e7d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2022-05-07T15:50:11.095 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.094+0000 7fe2c70e7d80 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2022-05-07T15:50:11.499 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:11.498+0000 7fe2c70e7d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:50:14.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/1/keyring 2022-05-07T15:50:14.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:50:14.499 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2022-05-07T15:50:14.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-05-07T15:50:14.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:50:14.905 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:50:14.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-05-07T15:50:14.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:14.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:50:14.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:50:14.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:50:14.931 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:14.930+0000 7f9ca2a5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:14.934 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:14.934+0000 7f9ca2a5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:14.936 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:14.934+0000 7f9ca2a5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:15.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-05-07T15:50:15.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:15.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:50:15.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:15.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:15.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:15.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:15.258 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:15.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:15.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:50:15.467 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:15.466+0000 7f9ca2a5cd80 -1 Falling back to public interface 2022-05-07T15:50:15.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:16.261 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:16.262+0000 7f9ca2a5cd80 -1 osd.1 0 log_to_monitors {default=true} 2022-05-07T15:50:16.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:16.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:16.571 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:16.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:50:16.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:16.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:50:16.934 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/12969,v1:127.0.0.1:6811/12969] [v2:127.0.0.1:6812/12969,v1:127.0.0.1:6813/12969] exists,up 77914f20-4f34-4d38-a7bd-57cea0b5dfb9 2022-05-07T15:50:16.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:16.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:16.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:16.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2022-05-07T15:50:16.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2022-05-07T15:50:16.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:50:16.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-05-07T15:50:16.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:50:16.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/2 2022-05-07T15:50:16.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 ' 2022-05-07T15:50:16.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:50:16.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:50:16.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:50:16.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2022-05-07T15:50:16.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2022-05-07T15:50:16.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:50:16.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:50:16.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2022-05-07T15:50:16.941 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:50:16.941 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:16.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:16.942 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:16.942 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:16.942 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:16.944 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:16.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' 2022-05-07T15:50:16.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:50:16.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:50:16.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:50:16.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2022-05-07T15:50:16.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2022-05-07T15:50:16.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:50:16.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:50:16.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:16.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:50:16.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:50:16.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/2 2022-05-07T15:50:16.950 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:50:16.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1b8bee36-79e8-4c37-8032-fb8f81a455a8 2022-05-07T15:50:16.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 1b8bee36-79e8-4c37-8032-fb8f81a455a8' 2022-05-07T15:50:16.955 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 1b8bee36-79e8-4c37-8032-fb8f81a455a8 2022-05-07T15:50:16.956 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:50:16.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA4lXZiAFMtOhAAegR3zdgFh9hLGywMf1hDYw== 2022-05-07T15:50:16.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA4lXZiAFMtOhAAegR3zdgFh9hLGywMf1hDYw=="}' 2022-05-07T15:50:16.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1b8bee36-79e8-4c37-8032-fb8f81a455a8 -i td/bad-inc-map/2/new.json 2022-05-07T15:50:17.440 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:50:17.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/2/new.json 2022-05-07T15:50:17.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA4lXZiAFMtOhAAegR3zdgFh9hLGywMf1hDYw== --osd-uuid 1b8bee36-79e8-4c37-8032-fb8f81a455a8 2022-05-07T15:50:17.474 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.474+0000 7fddeb300d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:17.477 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.478+0000 7fddeb300d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:17.478 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.478+0000 7fddeb300d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:17.479 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.478+0000 7fddeb300d80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2022-05-07T15:50:17.479 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.478+0000 7fddeb300d80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2022-05-07T15:50:17.480 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.478+0000 7fddeb300d80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2022-05-07T15:50:17.539 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:17.538+0000 7fddeb300d80 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2022-05-07T15:50:18.010 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:18.010+0000 7fddeb300d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:50:20.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/2/keyring 2022-05-07T15:50:20.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:50:20.895 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2022-05-07T15:50:20.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-05-07T15:50:20.896 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:50:21.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-05-07T15:50:21.308 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2022-05-07T15:50:21.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=e8557712-c75f-4070-a314-d774b9721dba --auth-supported=none --mon-host=127.0.0.1:45871 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.12171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:21.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:50:21.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:50:21.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:50:21.334 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:21.334+0000 7f34aa464d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:21.336 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:21.338+0000 7f34aa464d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:21.344 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:21.338+0000 7f34aa464d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:21.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-05-07T15:50:21.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:21.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:50:21.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:21.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:21.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:21.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:21.648 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:21.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:21.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:50:21.866 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:21.866+0000 7f34aa464d80 -1 Falling back to public interface 2022-05-07T15:50:21.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:22.712 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:22.710+0000 7f34aa464d80 -1 osd.2 0 log_to_monitors {default=true} 2022-05-07T15:50:22.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:22.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:22.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:50:22.963 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:22.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:22.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:50:23.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:24.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:24.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:24.376 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:50:24.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:50:24.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:24.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:50:24.696 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/13324,v1:127.0.0.1:6819/13324] [v2:127.0.0.1:6820/13324,v1:127.0.0.1:6821/13324] exists,up 1b8bee36-79e8-4c37-8032-fb8f81a455a8 2022-05-07T15:50:24.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:24.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:24.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:24.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2022-05-07T15:50:25.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2022-05-07T15:50:25.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 8 2022-05-07T15:50:25.712 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'foo' already exists 2022-05-07T15:50:25.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-05-07T15:50:26.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2022-05-07T15:50:27.536 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 1 2022-05-07T15:50:27.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2022-05-07T15:50:28.509 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2022-05-07T15:50:28.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2022-05-07T15:50:33.525 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2022-05-07T15:50:33.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2022-05-07T15:50:33.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:50:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:33.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:33.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2022-05-07T15:50:33.529 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:33.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:33.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:33.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:33.852 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/12523,v1:127.0.0.1:6801/12523] [v2:127.0.0.1:6802/12523,v1:127.0.0.1:6803/12523] exists,up cc7d238a-8931-4342-a38f-286f7e27d4d6 2022-05-07T15:50:33.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:33.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:33.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:33.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2022-05-07T15:50:33.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2022-05-07T15:50:33.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:33.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:50:33.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:33.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:33.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2022-05-07T15:50:33.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:33.856 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:33.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:33.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:50:34.167 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/12969,v1:127.0.0.1:6811/12969] [v2:127.0.0.1:6812/12969,v1:127.0.0.1:6813/12969] exists,up 77914f20-4f34-4d38-a7bd-57cea0b5dfb9 2022-05-07T15:50:34.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:34.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:34.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:34.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2022-05-07T15:50:34.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2022-05-07T15:50:34.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:34.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:50:34.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:34.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:34.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2022-05-07T15:50:34.172 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:34.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:34.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:34.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:50:34.479 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 14 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/13324,v1:127.0.0.1:6819/13324] [v2:127.0.0.1:6820/13324,v1:127.0.0.1:6821/13324] exists,up 1b8bee36-79e8-4c37-8032-fb8f81a455a8 2022-05-07T15:50:34.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:34.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:34.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:34.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2022-05-07T15:50:34.482 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:25.702+0000 7f349dd10700 -1 osd.2 16 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.482 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:26.042+0000 7f349dd10700 -1 osd.2 17 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.482 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:26.578+0000 7f349dd10700 -1 osd.2 18 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.483 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.318+0000 7f349b50b700 -1 osd.2 19 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.483 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.318+0000 7f349b50b700 -1 osd.2 19 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.483 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.670+0000 7f349b50b700 -1 osd.2 21 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.483 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.670+0000 7f349b50b700 -1 osd.2 21 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.483 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.810+0000 7f349dd10700 -1 osd.2 22 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.484 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.286+0000 7f349b50b700 -1 osd.2 23 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.484 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.286+0000 7f349b50b700 -1 osd.2 23 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.484 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.502+0000 7f349dd10700 -1 osd.2 24 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.485 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.802+0000 7f349dd10700 -1 osd.2 25 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.486 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.802+0000 7f349dd10700 -1 osd.2 25 handle_osd_map injecting map crc failure 2022-05-07T15:50:34.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2022-05-07T15:50:34.487 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:25.702+0000 7f349dd10700 10 osd.2 16 handle_osd_map bailing because last < start (16<17) 2022-05-07T15:50:34.487 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:26.042+0000 7f349dd10700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2022-05-07T15:50:34.487 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:26.578+0000 7f349dd10700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2022-05-07T15:50:34.488 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.318+0000 7f349b50b700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2022-05-07T15:50:34.488 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.322+0000 7f349b50b700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2022-05-07T15:50:34.488 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.670+0000 7f349b50b700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2022-05-07T15:50:34.488 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.674+0000 7f349b50b700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2022-05-07T15:50:34.489 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:27.814+0000 7f349dd10700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2022-05-07T15:50:34.489 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.286+0000 7f349b50b700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2022-05-07T15:50:34.489 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.286+0000 7f349b50b700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2022-05-07T15:50:34.489 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.502+0000 7f349dd10700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2022-05-07T15:50:34.489 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.802+0000 7f349dd10700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2022-05-07T15:50:34.490 INFO:tasks.workunit.client.0.smithi059.stdout:2022-05-07T15:50:28.806+0000 7f349dd10700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2022-05-07T15:50:34.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2022-05-07T15:50:34.491 INFO:tasks.workunit.client.0.smithi059.stdout:success 2022-05-07T15:50:34.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2022-05-07T15:50:34.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=foo 2022-05-07T15:50:34.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2022-05-07T15:50:35.021 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'foo' does not exist 2022-05-07T15:50:35.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2022-05-07T15:50:35.032 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:50:35.033 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:50:35.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:50:35.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:50:35.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:50:36.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:50:36.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2022-05-07T15:50:36.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2022-05-07T15:50:36.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:50:36.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2022-05-07T15:50:36.355 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:50:36.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:50:36.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:50:36.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:50:36.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:50:36.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:50:36.366 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:50:36.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:50:36.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:50:36.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:50:36.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:50:36.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:50:36.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:36.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:50:36.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:50:36.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:36.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:50:36.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:50:36.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2022-05-07T15:50:36.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:50:36.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:36.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:36.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12171 2022-05-07T15:50:36.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:50:36.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:50:36.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2022-05-07T15:50:36.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/bad-inc-map 0 2022-05-07T15:50:36.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2022-05-07T15:50:36.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2022-05-07T15:50:36.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2022-05-07T15:50:36.387 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:50:36.387 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:50:36.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:50:36.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:50:36.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:50:36.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:50:36.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:50:36.389 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:50:36.390 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:50:36.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:50:36.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:50:36.392 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:50:36.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:36.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:50:36.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:50:36.394 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:36.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:50:36.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2022-05-07T15:50:36.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2022-05-07T15:50:36.399 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:50:36.399 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:36.403 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12171 2022-05-07T15:50:36.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12171 2022-05-07T15:50:36.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:50:36.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:50:36.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2022-05-07T15:50:36.406 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-05-07T15:50:36.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-05-07T15:50:36.462 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2022-05-07T15:50:36.463 DEBUG:teuthology.orchestra.run.smithi059:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=63a8334b02d7dac9af12d1cf5f3ef0666b46826e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2022-05-07T15:50:36.482 INFO:tasks.workunit.client.0.smithi059.stderr:stty: 'standard input': Inappropriate ioctl for device 2022-05-07T15:50:36.498 INFO:tasks.workunit.client.0.smithi059.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2022-05-07T15:50:36.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2022-05-07T15:50:36.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2022-05-07T15:50:36.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2022-05-07T15:50:36.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2022-05-07T15:50:36.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2022-05-07T15:50:36.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2022-05-07T15:50:36.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2022-05-07T15:50:36.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2022-05-07T15:50:36.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/divergent-priors 2022-05-07T15:50:36.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2022-05-07T15:50:36.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2022-05-07T15:50:36.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2022-05-07T15:50:36.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2022-05-07T15:50:36.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2022-05-07T15:50:36.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2022-05-07T15:50:36.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2022-05-07T15:50:36.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2022-05-07T15:50:36.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2022-05-07T15:50:36.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2022-05-07T15:50:36.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2022-05-07T15:50:36.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2022-05-07T15:50:36.504 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2022-05-07T15:50:36.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2022-05-07T15:50:36.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2022-05-07T15:50:36.506 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2022-05-07T15:50:36.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2022-05-07T15:50:36.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2022-05-07T15:50:36.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2022-05-07T15:50:36.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2022-05-07T15:50:36.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2022-05-07T15:50:36.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2022-05-07T15:50:36.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2022-05-07T15:50:36.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2022-05-07T15:50:36.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2022-05-07T15:50:36.508 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2022-05-07T15:50:36.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none ' 2022-05-07T15:50:36.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2022-05-07T15:50:36.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2022-05-07T15:50:36.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2022-05-07T15:50:36.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:50:36.510 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2022-05-07T15:50:36.511 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2022-05-07T15:50:36.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2022-05-07T15:50:36.514 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_divergent_2 2022-05-07T15:50:36.514 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_divergent_3 2022-05-07T15:50:36.515 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_divergent_ec' 2022-05-07T15:50:36.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2022-05-07T15:50:36.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2022-05-07T15:50:36.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2022-05-07T15:50:36.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2022-05-07T15:50:36.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-05-07T15:50:36.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:50:36.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-05-07T15:50:36.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:50:36.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:50:36.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:50:36.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:50:36.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:50:36.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:50:36.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:50:36.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:50:36.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:50:36.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:50:36.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:50:36.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:50:36.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:36.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:50:36.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:50:36.524 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:50:36.525 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:50:36.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:50:36.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-05-07T15:50:36.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:50:36.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:36.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:36.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13828 2022-05-07T15:50:36.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:50:36.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:50:36.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2022-05-07T15:50:36.533 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-05-07T15:50:36.533 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:36.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:36.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.13828 2022-05-07T15:50:36.537 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-05-07T15:50:36.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2022-05-07T15:50:36.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2022-05-07T15:50:36.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-05-07T15:50:36.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2022-05-07T15:50:36.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2022-05-07T15:50:36.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2022-05-07T15:50:36.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2022-05-07T15:50:36.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2022-05-07T15:50:36.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2022-05-07T15:50:36.540 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2022-05-07T15:50:36.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2022-05-07T15:50:36.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2022-05-07T15:50:36.972 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:50:36.972 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:50:36.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2022-05-07T15:50:36.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2022-05-07T15:50:36.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-05-07T15:50:36.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-05-07T15:50:36.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-05-07T15:50:36.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2022-05-07T15:50:36.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2022-05-07T15:50:37.527 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-05-07T15:50:37.527 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:37.528 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:37.528 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:37.528 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:37.528 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:37.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:37.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-05-07T15:50:37.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-05-07T15:50:37.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-05-07T15:50:37.718 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:50:37.718 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:50:37.718 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-05-07T15:50:37.719 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-05-07T15:50:37.719 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:50:37.720 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:50:37.720 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:50:37.721 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:50:37.721 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:37.722 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:37.723 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-mon.a.asok 2022-05-07T15:50:37.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:50:37.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.13828/ceph-mon.a.asok config get fsid 2022-05-07T15:50:37.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-05-07T15:50:37.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:50:37.814 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:50:37.814 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-05-07T15:50:37.814 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-05-07T15:50:37.815 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:50:37.816 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:50:37.816 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:50:37.816 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:50:37.816 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:37.817 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:37.817 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-mon.a.asok 2022-05-07T15:50:37.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:50:37.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.13828/ceph-mon.a.asok config get mon_host 2022-05-07T15:50:37.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2022-05-07T15:50:37.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2022-05-07T15:50:37.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-05-07T15:50:37.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-05-07T15:50:37.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-05-07T15:50:37.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2022-05-07T15:50:37.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-05-07T15:50:38.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-05-07T15:50:38.799 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:38.799 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:38.800 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:38.801 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:38.801 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:38.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:38.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:50:38.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:50:38.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2022-05-07T15:50:38.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2022-05-07T15:50:38.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:50:38.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:50:38.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-05-07T15:50:38.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:50:38.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2022-05-07T15:50:38.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:50:38.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:50:38.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:50:38.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:50:38.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-05-07T15:50:38.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-05-07T15:50:38.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:50:38.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:50:38.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:50:38.844 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:50:38.845 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:38.845 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:38.847 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:38.847 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:38.847 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:38.848 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:38.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:38.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:50:38.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:50:38.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:50:38.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:50:38.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:50:38.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:50:38.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:50:38.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:38.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:50:38.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:50:38.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2022-05-07T15:50:38.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:50:38.852 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 2022-05-07T15:50:38.852 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 2022-05-07T15:50:38.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3' 2022-05-07T15:50:38.853 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:50:38.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBOlXZiW+AONBAAl2ibCKm7H5/2oiO5xUxABQ== 2022-05-07T15:50:38.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBOlXZiW+AONBAAl2ibCKm7H5/2oiO5xUxABQ=="}' 2022-05-07T15:50:38.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 -i td/divergent-priors/0/new.json 2022-05-07T15:50:39.193 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:39.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2022-05-07T15:50:39.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOlXZiW+AONBAAl2ibCKm7H5/2oiO5xUxABQ== --osd-uuid cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 2022-05-07T15:50:39.242 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.238+0000 7f5c62072d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:39.242 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.242+0000 7f5c62072d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:39.244 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.242+0000 7f5c62072d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:39.245 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.246+0000 7f5c62072d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:50:39.245 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.246+0000 7f5c62072d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:50:39.245 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.246+0000 7f5c62072d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:50:39.282 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.282+0000 7f5c62072d80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2022-05-07T15:50:39.684 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:39.682+0000 7f5c62072d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:50:42.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2022-05-07T15:50:42.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:50:42.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-05-07T15:50:42.443 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2022-05-07T15:50:42.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:50:42.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-05-07T15:50:42.887 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2022-05-07T15:50:42.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:42.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:50:42.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:50:42.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:50:42.924 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:42.922+0000 7f0271079d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:42.925 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:42.926+0000 7f0271079d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:42.927 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:42.926+0000 7f0271079d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:43.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-05-07T15:50:43.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:43.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:50:43.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:43.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:43.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:43.206 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:43.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:43.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:43.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:43.463 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:43.462+0000 7f0271079d80 -1 Falling back to public interface 2022-05-07T15:50:43.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:44.280 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:44.278+0000 7f0271079d80 -1 osd.0 0 log_to_monitors {default=true} 2022-05-07T15:50:44.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:44.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:44.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:50:44.534 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:44.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:44.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:44.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:45.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:45.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:45.915 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:50:45.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:50:45.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:45.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:50:46.246 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/14245,v1:127.0.0.1:6803/14245] [v2:127.0.0.1:6804/14245,v1:127.0.0.1:6805/14245] exists,up cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 2022-05-07T15:50:46.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:46.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:46.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:46.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2022-05-07T15:50:46.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2022-05-07T15:50:46.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:50:46.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:50:46.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-05-07T15:50:46.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:50:46.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:50:46.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:50:46.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:50:46.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:50:46.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:50:46.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:50:46.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:50:46.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:50:46.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:50:46.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:50:46.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:50:46.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:46.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:46.255 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:46.255 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:46.256 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:46.256 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:46.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:46.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:50:46.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:50:46.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:50:46.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:50:46.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:50:46.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:50:46.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:50:46.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:46.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:50:46.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:50:46.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:50:46.260 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:50:46.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=56846436-68b9-4d7a-82ba-c1f770c44ac1 2022-05-07T15:50:46.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 56846436-68b9-4d7a-82ba-c1f770c44ac1' 2022-05-07T15:50:46.261 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 56846436-68b9-4d7a-82ba-c1f770c44ac1 2022-05-07T15:50:46.261 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:50:46.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBWlXZigLq4EBAAFp/OwTU3gTpBoRgW5IPdXA== 2022-05-07T15:50:46.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBWlXZigLq4EBAAFp/OwTU3gTpBoRgW5IPdXA=="}' 2022-05-07T15:50:46.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 56846436-68b9-4d7a-82ba-c1f770c44ac1 -i td/divergent-priors/1/new.json 2022-05-07T15:50:46.712 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:46.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2022-05-07T15:50:46.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWlXZigLq4EBAAFp/OwTU3gTpBoRgW5IPdXA== --osd-uuid 56846436-68b9-4d7a-82ba-c1f770c44ac1 2022-05-07T15:50:46.762 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.762+0000 7f648588dd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:46.765 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.766+0000 7f648588dd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:46.767 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.766+0000 7f648588dd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:46.767 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.766+0000 7f648588dd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:50:46.768 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.766+0000 7f648588dd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:50:46.768 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.766+0000 7f648588dd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:50:46.808 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:46.806+0000 7f648588dd80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2022-05-07T15:50:47.269 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:47.270+0000 7f648588dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:50:50.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2022-05-07T15:50:50.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:50:50.084 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2022-05-07T15:50:50.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-05-07T15:50:50.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:50:50.512 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-05-07T15:50:50.512 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:50:50.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:50:50.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:50.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:50:50.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:50:50.544 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:50.546+0000 7f60a5720d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:50.546 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:50.546+0000 7f60a5720d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:50.548 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:50.550+0000 7f60a5720d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:50.847 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:50.846+0000 7f60a5720d80 -1 Falling back to public interface 2022-05-07T15:50:50.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-05-07T15:50:50.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:50.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:50:50.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:50.852 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:50.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:50.858 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:50.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:50.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:50.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:50:51.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:51.657 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:51.658+0000 7f60a5720d80 -1 osd.1 0 log_to_monitors {default=true} 2022-05-07T15:50:52.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:52.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:52.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:50:52.216 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:52.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:52.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:50:52.553 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/14628,v1:127.0.0.1:6811/14628] [v2:127.0.0.1:6812/14628,v1:127.0.0.1:6813/14628] exists,up 56846436-68b9-4d7a-82ba-c1f770c44ac1 2022-05-07T15:50:52.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:50:52.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:50:52.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:50:52.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2022-05-07T15:50:52.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2022-05-07T15:50:52.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:50:52.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:50:52.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-05-07T15:50:52.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:50:52.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2022-05-07T15:50:52.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:50:52.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:50:52.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:50:52.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:50:52.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-05-07T15:50:52.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-05-07T15:50:52.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:50:52.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:50:52.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:50:52.559 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:50:52.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:50:52.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:50:52.562 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:50:52.563 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:50:52.563 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:50:52.563 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:52.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:50:52.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:50:52.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:50:52.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:50:52.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:50:52.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:50:52.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:50:52.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:50:52.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:52.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:50:52.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:50:52.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2022-05-07T15:50:52.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:50:52.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6c079b36-581d-4b23-8fed-4f24d6c517ec 2022-05-07T15:50:52.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 6c079b36-581d-4b23-8fed-4f24d6c517ec' 2022-05-07T15:50:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 6c079b36-581d-4b23-8fed-4f24d6c517ec 2022-05-07T15:50:52.567 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:50:52.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBclXZia1v0IhAA9tbTIeg5362DN4fTLwRCmg== 2022-05-07T15:50:52.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBclXZia1v0IhAA9tbTIeg5362DN4fTLwRCmg=="}' 2022-05-07T15:50:52.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6c079b36-581d-4b23-8fed-4f24d6c517ec -i td/divergent-priors/2/new.json 2022-05-07T15:50:53.108 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:50:53.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2022-05-07T15:50:53.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBclXZia1v0IhAA9tbTIeg5362DN4fTLwRCmg== --osd-uuid 6c079b36-581d-4b23-8fed-4f24d6c517ec 2022-05-07T15:50:53.161 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.162+0000 7f325a991d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:53.164 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.166+0000 7f325a991d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:53.166 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.166+0000 7f325a991d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:53.166 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.166+0000 7f325a991d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:50:53.167 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.166+0000 7f325a991d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:50:53.167 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.166+0000 7f325a991d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:50:53.210 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.210+0000 7f325a991d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2022-05-07T15:50:53.629 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:53.630+0000 7f325a991d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:50:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2022-05-07T15:50:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:50:56.337 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2022-05-07T15:50:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-05-07T15:50:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:50:56.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-05-07T15:50:56.761 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2022-05-07T15:50:56.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:50:56.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:50:56.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:50:56.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:50:56.795 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:56.794+0000 7f89a33b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:56.797 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:56.798+0000 7f89a33b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:56.799 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:56.798+0000 7f89a33b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:50:57.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-05-07T15:50:57.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:50:57.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:50:57.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:50:57.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:50:57.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:57.129 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:50:57.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:50:57.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:57.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:50:57.370 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:57.370+0000 7f89a33b1d80 -1 Falling back to public interface 2022-05-07T15:50:57.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:58.235 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:50:58.234+0000 7f89a33b1d80 -1 osd.2 0 log_to_monitors {default=true} 2022-05-07T15:50:58.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:58.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:58.451 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:50:58.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:50:58.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:58.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:50:58.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:50:59.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:50:59.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:50:59.883 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:50:59.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:50:59.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:50:59.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:51:00.237 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/14983,v1:127.0.0.1:6819/14983] [v2:127.0.0.1:6820/14983,v1:127.0.0.1:6821/14983] exists,up 6c079b36-581d-4b23-8fed-4f24d6c517ec 2022-05-07T15:51:00.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:51:00.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:51:00.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:51:00.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2022-05-07T15:51:00.857 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2022-05-07T15:51:00.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2022-05-07T15:51:01.526 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2022-05-07T15:51:01.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2022-05-07T15:51:02.244 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2022-05-07T15:51:02.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2022-05-07T15:51:02.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2022-05-07T15:51:02.761 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2022-05-07T15:51:02.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-05-07T15:51:03.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2022-05-07T15:51:04.469 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 3 2022-05-07T15:51:04.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2022-05-07T15:51:05.187 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2022-05-07T15:51:05.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2022-05-07T15:51:05.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:51:05.199 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:51:05.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:51:05.548 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:51:05.548 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:51:05.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:51:05.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:05.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:51:05.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836488 2022-05-07T15:51:05.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836488 2022-05-07T15:51:05.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488' 2022-05-07T15:51:05.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:05.718 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:51:05.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2022-05-07T15:51:05.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2022-05-07T15:51:05.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2022-05-07T15:51:05.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:05.873 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:51:06.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2022-05-07T15:51:06.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2022-05-07T15:51:06.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-64424509444' 2022-05-07T15:51:06.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:06.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836488 2022-05-07T15:51:06.027 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:06.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:51:06.029 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2022-05-07T15:51:06.029 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:06.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836488 2022-05-07T15:51:06.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2022-05-07T15:51:06.032 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836488 2022-05-07T15:51:06.032 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:06.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836488 2022-05-07T15:51:06.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:06.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2022-05-07T15:51:06.369 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:06.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:51:06.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2022-05-07T15:51:06.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:06.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2022-05-07T15:51:06.373 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2022-05-07T15:51:06.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2022-05-07T15:51:06.374 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:51:06.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2022-05-07T15:51:06.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:06.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2022-05-07T15:51:06.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:06.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:51:06.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2022-05-07T15:51:06.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:06.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2022-05-07T15:51:06.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2022-05-07T15:51:06.750 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509444 2022-05-07T15:51:06.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:51:07.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509442 -lt 64424509444 2022-05-07T15:51:07.131 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:51:08.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:51:08.133 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:51:08.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2022-05-07T15:51:08.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2022-05-07T15:51:08.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:51:08.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:51:08.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:51:08.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:51:08.474 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2022-05-07T15:51:08.475 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:51:08.475 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:51:08.475 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:51:08.475 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:51:08.476 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:51:08.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:51:08.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:51:08.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:51:08.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:51:08.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:51:09.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:51:09.011 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:51:09.011 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:51:09.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:51:09.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:09.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:51:09.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2022-05-07T15:51:09.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2022-05-07T15:51:09.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2022-05-07T15:51:09.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:09.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:51:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2022-05-07T15:51:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2022-05-07T15:51:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2022-05-07T15:51:09.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:09.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:51:09.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2022-05-07T15:51:09.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2022-05-07T15:51:09.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509446' 2022-05-07T15:51:09.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:09.450 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2022-05-07T15:51:09.450 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:09.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:51:09.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2022-05-07T15:51:09.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:09.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2022-05-07T15:51:09.454 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836490 2022-05-07T15:51:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2022-05-07T15:51:09.455 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:09.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836490 2022-05-07T15:51:09.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:51:10.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:51:10.788 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:11.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2022-05-07T15:51:11.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:11.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2022-05-07T15:51:11.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:11.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:51:11.170 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2022-05-07T15:51:11.170 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:11.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2022-05-07T15:51:11.172 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672968 2022-05-07T15:51:11.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2022-05-07T15:51:11.173 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:51:11.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2022-05-07T15:51:11.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:11.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2022-05-07T15:51:11.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:11.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:51:11.537 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2022-05-07T15:51:11.537 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:11.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2022-05-07T15:51:11.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2022-05-07T15:51:11.540 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509446 2022-05-07T15:51:11.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:51:11.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509446 2022-05-07T15:51:11.906 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:51:11.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:11.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:12.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:51:12.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:51:12.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:51:12.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:51:12.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:51:12.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:51:12.318 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:51:12.318 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:51:12.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-05-07T15:51:12.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:51:12.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:12.645 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:13.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-05-07T15:51:13.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-05-07T15:51:13.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-05-07T15:51:13.054 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2022-05-07T15:51:13.054 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2022-05-07T15:51:13.380 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:51:13.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2022-05-07T15:51:13.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2022-05-07T15:51:13.389 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2022-05-07T15:51:13.389 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2022-05-07T15:51:13.700 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:51:13.700 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-05-07T15:51:03.462786+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:51:13.701 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:51:13.701 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:51:13.703 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:51:13.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2022-05-07T15:51:13.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2022-05-07T15:51:13.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2022-05-07T15:51:13.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2022-05-07T15:51:13.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2022-05-07T15:51:13.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2022-05-07T15:51:13.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2022-05-07T15:51:13.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2022-05-07T15:51:13.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2022-05-07T15:51:13.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2022-05-07T15:51:13.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2022-05-07T15:51:13.714 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2022-05-07T15:51:13.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2022-05-07T15:51:13.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:13.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2022-05-07T15:51:13.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:13.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2022-05-07T15:51:13.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:13.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2022-05-07T15:51:14.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:14.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2022-05-07T15:51:14.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:14.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2022-05-07T15:51:14.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:14.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2022-05-07T15:51:14.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:14.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2022-05-07T15:51:14.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:14.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2022-05-07T15:51:14.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:14.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2022-05-07T15:51:15.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2022-05-07T15:51:15.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2022-05-07T15:51:15.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2022-05-07T15:51:15.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2022-05-07T15:51:15.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2022-05-07T15:51:15.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2022-05-07T15:51:15.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:15.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2022-05-07T15:51:16.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2022-05-07T15:51:16.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2022-05-07T15:51:16.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2022-05-07T15:51:16.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2022-05-07T15:51:16.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2022-05-07T15:51:16.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2022-05-07T15:51:16.880 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:16.880 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2022-05-07T15:51:17.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2022-05-07T15:51:17.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2022-05-07T15:51:17.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2022-05-07T15:51:17.424 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.424 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2022-05-07T15:51:17.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2022-05-07T15:51:17.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2022-05-07T15:51:17.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:17.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2022-05-07T15:51:18.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2022-05-07T15:51:18.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2022-05-07T15:51:18.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2022-05-07T15:51:18.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2022-05-07T15:51:18.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2022-05-07T15:51:18.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2022-05-07T15:51:18.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:18.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2022-05-07T15:51:19.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2022-05-07T15:51:19.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2022-05-07T15:51:19.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2022-05-07T15:51:19.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2022-05-07T15:51:19.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2022-05-07T15:51:19.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2022-05-07T15:51:19.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:19.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2022-05-07T15:51:20.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2022-05-07T15:51:20.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.189 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2022-05-07T15:51:20.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2022-05-07T15:51:20.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2022-05-07T15:51:20.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2022-05-07T15:51:20.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2022-05-07T15:51:20.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:20.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2022-05-07T15:51:21.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2022-05-07T15:51:21.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2022-05-07T15:51:21.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2022-05-07T15:51:21.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2022-05-07T15:51:21.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2022-05-07T15:51:21.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2022-05-07T15:51:21.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:21.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2022-05-07T15:51:22.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2022-05-07T15:51:22.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2022-05-07T15:51:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2022-05-07T15:51:22.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2022-05-07T15:51:22.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2022-05-07T15:51:22.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2022-05-07T15:51:22.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:22.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2022-05-07T15:51:23.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2022-05-07T15:51:23.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2022-05-07T15:51:23.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2022-05-07T15:51:23.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2022-05-07T15:51:23.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2022-05-07T15:51:23.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2022-05-07T15:51:23.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:23.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2022-05-07T15:51:24.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2022-05-07T15:51:24.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.158 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2022-05-07T15:51:24.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2022-05-07T15:51:24.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2022-05-07T15:51:24.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2022-05-07T15:51:24.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.770 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2022-05-07T15:51:24.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:24.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2022-05-07T15:51:25.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2022-05-07T15:51:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2022-05-07T15:51:25.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2022-05-07T15:51:25.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2022-05-07T15:51:25.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2022-05-07T15:51:25.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2022-05-07T15:51:25.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:25.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2022-05-07T15:51:26.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2022-05-07T15:51:26.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2022-05-07T15:51:26.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2022-05-07T15:51:26.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2022-05-07T15:51:26.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2022-05-07T15:51:26.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2022-05-07T15:51:26.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:26.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2022-05-07T15:51:27.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2022-05-07T15:51:27.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2022-05-07T15:51:27.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2022-05-07T15:51:27.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2022-05-07T15:51:27.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2022-05-07T15:51:27.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2022-05-07T15:51:27.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-05-07T15:51:27.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2022-05-07T15:51:28.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2022-05-07T15:51:28.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2022-05-07T15:51:28.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:51:28.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:51:28.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:51:28.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:51:28.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-05-07T15:51:28.096 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:51:28.096 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:51:28.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:51:28.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:51:28.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:51:28.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:51:28.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:51:28.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:51:28.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:51:28.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:51:28.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:51:28.576 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:51:28.577 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:51:28.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:51:28.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:28.578 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:51:28.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2022-05-07T15:51:28.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2022-05-07T15:51:28.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2022-05-07T15:51:28.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:28.732 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:51:28.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2022-05-07T15:51:28.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2022-05-07T15:51:28.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2022-05-07T15:51:28.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:28.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:51:29.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2022-05-07T15:51:29.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2022-05-07T15:51:29.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2022-05-07T15:51:29.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:29.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2022-05-07T15:51:29.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:29.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:51:29.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2022-05-07T15:51:29.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:29.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2022-05-07T15:51:29.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2022-05-07T15:51:29.052 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836496 2022-05-07T15:51:29.052 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:29.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836496 2022-05-07T15:51:29.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:51:30.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:51:30.408 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:30.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2022-05-07T15:51:30.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:30.775 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2022-05-07T15:51:30.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:30.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:51:30.779 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2022-05-07T15:51:30.779 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:30.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2022-05-07T15:51:30.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2022-05-07T15:51:30.782 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672974 2022-05-07T15:51:30.782 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:51:31.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2022-05-07T15:51:31.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:31.107 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2022-05-07T15:51:31.107 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:31.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:51:31.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2022-05-07T15:51:31.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:31.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2022-05-07T15:51:31.112 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509452 2022-05-07T15:51:31.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2022-05-07T15:51:31.113 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:51:31.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509453 -lt 64424509452 2022-05-07T15:51:31.436 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:51:31.436 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:31.437 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:31.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:51:31.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:51:31.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:51:31.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:51:31.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:51:31.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:51:31.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:51:31.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:51:32.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-05-07T15:51:32.189 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:51:32.190 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:32.190 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:32.593 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-05-07T15:51:32.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-05-07T15:51:32.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-05-07T15:51:32.594 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2022-05-07T15:51:32.595 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2022-05-07T15:51:32.595 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2022-05-07T15:51:32.595 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2022-05-07T15:51:32.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2022-05-07T15:51:32.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2022-05-07T15:51:32.940 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2022-05-07T15:51:32.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2022-05-07T15:51:32.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2022-05-07T15:51:33.254 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:51:33.254 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2022-05-07T15:51:03.462786+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:51:33.254 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:51:33.255 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:51:33.257 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:51:33.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:33.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2022-05-07T15:51:33.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-05-07T15:51:33.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-05-07T15:51:33.267 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:51:33.268 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:51:33.268 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:51:33.268 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.0.asok 2022-05-07T15:51:33.268 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2022-05-07T15:51:33.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.0.asok config set objectstore_blackhole 1 2022-05-07T15:51:33.342 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:51:33.342 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-05-07T15:51:33.343 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:51:33.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:33.350 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2022-05-07T15:51:33.350 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-05-07T15:51:33.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-05-07T15:51:33.352 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:51:33.352 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:51:33.352 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:51:33.352 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.2.asok 2022-05-07T15:51:33.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2022-05-07T15:51:33.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.2.asok config set objectstore_blackhole 1 2022-05-07T15:51:33.434 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:51:33.434 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-05-07T15:51:33.435 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:51:33.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2022-05-07T15:51:33.443 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2022-05-07T15:51:33.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2022-05-07T15:51:33.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2022-05-07T15:51:33.445 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent object 2022-05-07T15:51:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2022-05-07T15:51:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2022-05-07T15:51:33.446 INFO:tasks.workunit.client.0.smithi059.stdout:create missing divergent object 2022-05-07T15:51:33.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2022-05-07T15:51:33.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2022-05-07T15:51:33.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2022-05-07T15:51:33.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2022-05-07T15:51:33.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2022-05-07T15:51:33.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test 2022-05-07T15:51:33.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2022-05-07T15:51:33.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=existing_99 2022-05-07T15:51:33.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2022-05-07T15:51:33.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/divergent-priors 2022-05-07T15:51:33.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2022-05-07T15:51:33.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2022-05-07T15:51:33.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2022-05-07T15:51:33.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2022-05-07T15:51:33.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test existing_99 2022-05-07T15:51:33.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2022-05-07T15:51:33.454 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=existing_99 2022-05-07T15:51:33.454 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test existing_99 2022-05-07T15:51:33.454 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2022-05-07T15:51:33.800 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2022-05-07T15:51:33.800 INFO:tasks.workunit.client.0.smithi059.stderr:0 2022-05-07T15:51:33.800 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:51:33.800 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2022-05-07T15:51:33.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2022-05-07T15:51:33.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2022-05-07T15:51:33.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2022-05-07T15:51:33.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2022-05-07T15:51:33.803 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/divergent-priors/1/type 2022-05-07T15:51:33.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2022-05-07T15:51:33.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2022-05-07T15:51:33.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2022-05-07T15:51:33.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2022-05-07T15:51:33.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2022-05-07T15:51:33.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2022-05-07T15:51:33.807 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2022-05-07T15:51:33.808 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2022-05-07T15:51:33.808 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-05-07T15:51:33.808 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-05-07T15:51:33.808 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:51:33.809 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:51:33.809 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:51:33.809 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.1.asok 2022-05-07T15:51:33.809 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.13828/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2022-05-07T15:51:33.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2022-05-07T15:51:33.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2022-05-07T15:51:33.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2022-05-07T15:51:33.899 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2022-05-07T15:51:33.899 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-05-07T15:51:33.899 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-05-07T15:51:33.900 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:51:33.900 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:51:33.900 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:51:33.901 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.1.asok 2022-05-07T15:51:33.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2022-05-07T15:51:33.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.1.asok injectdataerr test existing_99 2022-05-07T15:51:33.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2022-05-07T15:51:33.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2022-05-07T15:51:43.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2022-05-07T15:51:44.003 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2022-05-07T15:51:44.003 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2022-05-07T15:51:44.004 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2022-05-07T15:51:44.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2022-05-07T15:51:44.327 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:51:44.328 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 active+clean 2022-05-07T15:51:34.306791+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:51:44.328 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:51:44.328 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:51:44.330 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:51:44.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2022-05-07T15:51:44.339 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:51:44.339 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:51:44.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:51:44.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:51:44.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:51:44.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:51:44.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2022-05-07T15:51:44.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2022-05-07T15:51:44.926 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already down. 2022-05-07T15:51:44.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2022-05-07T15:51:44.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2022-05-07T15:51:45.410 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already down. 2022-05-07T15:51:45.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2022-05-07T15:51:45.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2022-05-07T15:51:45.894 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already down. 2022-05-07T15:51:45.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:45.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2022-05-07T15:51:46.435 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2022-05-07T15:51:46.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:46.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2022-05-07T15:51:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already out. 2022-05-07T15:51:46.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2022-05-07T15:51:46.953 INFO:tasks.workunit.client.0.smithi059.stdout:bringing up non_divergent 0 2 2022-05-07T15:51:46.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2022-05-07T15:51:47.264 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:51:47.442 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 stale+active+clean 2022-05-07T15:51:34.306791+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:51:47.442 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:51:47.443 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:51:47.447 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:51:47.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:47.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2022-05-07T15:51:47.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:51:47.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:51:47.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2022-05-07T15:51:47.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:51:47.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2022-05-07T15:51:47.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:51:47.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:51:47.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:51:47.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:51:47.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-05-07T15:51:47.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-05-07T15:51:47.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:51:47.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:51:47.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:51:47.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:51:47.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:51:47.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:51:47.452 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:51:47.452 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:51:47.452 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:51:47.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:51:47.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:51:47.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:51:47.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:51:47.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:51:47.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:51:47.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:51:47.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:51:47.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:51:47.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:51:47.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2022-05-07T15:51:47.455 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2022-05-07T15:51:47.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2022-05-07T15:51:47.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:51:47.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2022-05-07T15:51:47.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2022-05-07T15:51:47.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:51:47.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:51:47.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:51:47.459 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:47.315+0000 7fa14bc23d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:51:47.459 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:47.319+0000 7fa14bc23d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:51:47.459 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:47.319+0000 7fa14bc23d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:51:47.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2022-05-07T15:51:47.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:51:47.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:51:47.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:51:47.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:51:47.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:51:47.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:51:47.615 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:51:47.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:51:47.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:51:47.848 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:47.847+0000 7fa14bc23d80 -1 Falling back to public interface 2022-05-07T15:51:47.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:51:48.787 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:48.787+0000 7fa14bc23d80 -1 osd.0 29 log_to_monitors {default=true} 2022-05-07T15:51:48.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:51:48.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:51:48.953 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:51:48.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:51:48.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:51:48.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:51:49.372 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/19006,v1:127.0.0.1:6803/19006] [v2:127.0.0.1:6804/19006,v1:127.0.0.1:6805/19006] exists,up cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 2022-05-07T15:51:49.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:51:49.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:51:49.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:51:49.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:49.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2022-05-07T15:51:49.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:51:49.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:51:49.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2022-05-07T15:51:49.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:51:49.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2022-05-07T15:51:49.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:51:49.380 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:51:49.380 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:51:49.380 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:51:49.380 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-05-07T15:51:49.381 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-05-07T15:51:49.381 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:51:49.381 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:51:49.381 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:51:49.382 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:51:49.382 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:51:49.382 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:51:49.382 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:51:49.383 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:51:49.383 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:51:49.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:51:49.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:51:49.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:51:49.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:51:49.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:51:49.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:51:49.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:51:49.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:51:49.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:51:49.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:51:49.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2022-05-07T15:51:49.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2022-05-07T15:51:49.386 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2022-05-07T15:51:49.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:51:49.388 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2022-05-07T15:51:49.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2022-05-07T15:51:49.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:51:49.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:51:49.389 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:51:49.408 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:49.407+0000 7f93cc4fad80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:51:49.411 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:49.411+0000 7f93cc4fad80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:51:49.418 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:49.411+0000 7f93cc4fad80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:51:49.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2022-05-07T15:51:49.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:51:49.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:51:49.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:51:49.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:51:49.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:51:49.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:51:49.720 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:51:49.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:51:49.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:51:49.944 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:49.943+0000 7f93cc4fad80 -1 Falling back to public interface 2022-05-07T15:51:50.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:51:50.866 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:51:50.867+0000 7f93cc4fad80 -1 osd.2 29 log_to_monitors {default=true} 2022-05-07T15:51:51.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:51:51.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:51:51.048 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:51:51.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:51:51.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:51:51.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:51:51.384 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/19200,v1:127.0.0.1:6811/19200] [v2:127.0.0.1:6812/19200,v1:127.0.0.1:6813/19200] exists,up 6c079b36-581d-4b23-8fed-4f24d6c517ec 2022-05-07T15:51:51.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:51:51.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:51:51.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:51:51.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:51.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2022-05-07T15:51:51.921 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already in. 2022-05-07T15:51:51.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2022-05-07T15:51:51.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2022-05-07T15:51:52.417 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already in. 2022-05-07T15:51:52.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2022-05-07T15:51:52.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2022-05-07T15:51:52.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:51:52.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:51:52.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:51:52.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:51:52.427 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-05-07T15:51:52.428 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:51:52.428 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:51:52.428 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:51:52.428 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:51:52.429 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:51:52.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:51:52.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:51:52.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:51:52.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:51:52.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:51:52.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:51:52.926 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:51:52.927 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:51:52.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:51:52.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:52.927 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:51:53.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2022-05-07T15:51:53.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2022-05-07T15:51:53.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822659' 2022-05-07T15:51:53.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:53.068 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:51:53.211 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-05-07T15:51:53.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-05-07T15:51:53.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-05-07T15:51:53.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-05-07T15:51:53.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:51:53.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:51:53.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757251 2022-05-07T15:51:53.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757251 2022-05-07T15:51:53.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822659 2-163208757251' 2022-05-07T15:51:53.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:53.366 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-154618822659 2022-05-07T15:51:53.366 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:53.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:51:53.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822659 2022-05-07T15:51:53.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:53.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2022-05-07T15:51:53.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2022-05-07T15:51:53.371 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 154618822659 2022-05-07T15:51:53.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:53.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 154618822659 2022-05-07T15:51:53.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:51:54.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:51:54.728 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:51:55.071 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822660 -lt 154618822659 2022-05-07T15:51:55.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:51:55.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-163208757251 2022-05-07T15:51:55.073 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:51:55.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:51:55.075 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:51:55.076 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-163208757251 2022-05-07T15:51:55.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757251 2022-05-07T15:51:55.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2022-05-07T15:51:55.078 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 163208757251 2022-05-07T15:51:55.078 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:51:55.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757251 -lt 163208757251 2022-05-07T15:51:55.417 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:51:55.418 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:55.418 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:55.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:51:55.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:51:55.834 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:51:55.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:51:55.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:51:55.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:51:55.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:51:55.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:51:56.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:51:56.173 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:51:56.173 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:56.173 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:56.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:51:56.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2022-05-07T15:51:56.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2022-05-07T15:51:56.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2022-05-07T15:51:56.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:51:56.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2022-05-07T15:51:56.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:51:56.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:51:56.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:51:56.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:51:56.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:51:56.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:51:56.685 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:51:56.685 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:51:57.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:51:57.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:51:57.008 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:57.008 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:57.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:51:57.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:51:57.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:51:57.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:51:57.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:51:57.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:51:57.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:51:57.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:51:57.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:51:57.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:51:57.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:51:57.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2022-05-07T15:51:57.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:51:57.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2022-05-07T15:51:58.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:51:58.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:51:58.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:51:58.013 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:51:58.013 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:51:58.013 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:51:58.013 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:51:58.014 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:51:58.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:51:58.328 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:51:58.328 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:58.329 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:51:58.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:51:58.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:51:58.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:51:58.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:51:58.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:51:58.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:51:58.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:51:58.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:51:58.724 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:51:59.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:51:59.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:51:59.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2022-05-07T15:51:59.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:51:59.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2022-05-07T15:51:59.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:51:59.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:51:59.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:51:59.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:51:59.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:51:59.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:51:59.577 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:51:59.577 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:51:59.912 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:51:59.913 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:51:59.914 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:51:59.914 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:52:00.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:52:00.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:52:00.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:52:00.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:52:00.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:52:00.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:52:00.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:52:00.328 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:52:00.328 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:52:00.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:52:00.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:52:00.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2022-05-07T15:52:00.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:52:00.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2022-05-07T15:52:01.525 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:52:01.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:52:01.527 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:52:01.527 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:52:01.528 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:52:01.528 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:52:01.528 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:52:01.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:52:01.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:52:01.853 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:52:01.854 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:52:01.854 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:52:02.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:52:02.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:52:02.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:52:02.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:52:02.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:52:02.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:52:02.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:52:02.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:52:02.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:52:02.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:52:02.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:52:02.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2022-05-07T15:52:02.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:52:02.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2022-05-07T15:52:04.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:52:04.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:52:04.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:52:04.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:52:04.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:52:04.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:52:04.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:52:04.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:52:04.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:52:04.631 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:52:04.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:52:04.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:52:05.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:52:05.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:52:05.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:52:05.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:52:05.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:52:05.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:52:05.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:52:05.119 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:52:05.119 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:52:05.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:52:05.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:52:05.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2022-05-07T15:52:05.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:52:05.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2022-05-07T15:52:08.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:52:08.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:52:08.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:52:08.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:52:08.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:52:08.726 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:52:08.726 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:52:08.726 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:52:09.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:52:09.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:52:09.059 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:52:09.059 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:52:09.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:52:09.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:52:09.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:52:09.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:52:09.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:52:09.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:52:09.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:52:09.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:52:09.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:52:09.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:52:09.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:52:09.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2022-05-07T15:52:09.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:52:09.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2022-05-07T15:52:16.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:52:16.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:52:16.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:52:16.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:52:16.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:52:16.273 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:52:16.273 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:52:16.273 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:52:16.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:52:16.605 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:52:16.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:52:16.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:52:17.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:52:17.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:52:17.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:52:17.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:52:17.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:52:17.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:52:17.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:52:17.018 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:52:17.018 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:52:17.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:52:17.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:52:17.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2022-05-07T15:52:17.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:52:17.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2022-05-07T15:52:24.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:52:24.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:52:24.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:52:24.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:52:24.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:52:24.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:52:24.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:52:24.735 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:52:25.066 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:52:25.066 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:52:25.067 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:52:25.067 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:52:25.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:52:25.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:52:25.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:52:25.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:52:25.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:52:25.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:52:25.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:52:25.486 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:52:25.487 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:52:25.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:52:25.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:52:25.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2022-05-07T15:52:25.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2022-05-07T15:52:26.298 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:52:26.299 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "5798dfe9-0064-4265-8506-4c35ea1664dd", 2022-05-07T15:52:26.299 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.2.16-103-g63a8334b", 2022-05-07T15:52:26.299 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "63a8334b02d7dac9af12d1cf5f3ef0666b46826e", 2022-05-07T15:52:26.299 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2022-05-07T15:52:26.300184+0000", 2022-05-07T15:52:26.300 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2022-05-07T15:52:26.300 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2022-05-07T15:52:26.300 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2022-05-07T15:52:26.300 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2022-05-07T15:52:26.301 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2022-05-07T15:52:26.301 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:52:26.301 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:52:26.301 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2022-05-07T15:52:26.302 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2022-05-07T15:52:26.302 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.302 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2022-05-07T15:52:26.302 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:52:26.303 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.303 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2022-05-07T15:52:26.303 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:52:26.303 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:52:26.304 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2022-05-07T15:52:26.304 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:52:26.304 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.304 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:52:26.305 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.305 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2022-05-07T15:52:26.305 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.305 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.306 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:52:26.306 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.306 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2022-05-07T15:52:26.306 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:52:26.307 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:52:26.307 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2022-05-07T15:52:26.307 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:52:26.307 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.308 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:52:26.308 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.308 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2022-05-07T15:52:26.308 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.309 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.309 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:52:26.309 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.309 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2022-05-07T15:52:26.310 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:52:26.310 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:52:26.310 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2022-05-07T15:52:26.310 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:52:26.311 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.311 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:52:26.311 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.312 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2022-05-07T15:52:26.312 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.312 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.312 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2022-05-07T15:52:26.313 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.313 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.313 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:52:26.313 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.314 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.314 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2022-05-07T15:52:26.314 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.314 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2022-05-07T15:52:26.315 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2022-05-07T15:52:26.315 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2022-05-07T15:52:26.315 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2022-05-07T15:52:26.315 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "c67bfd60-15c3-4eb2-a02f-223bbda3f2bb", 2022-05-07T15:52:26.316 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:50:37.005203Z", 2022-05-07T15:52:26.316 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2022-05-07T15:50:37.005203Z", 2022-05-07T15:52:26.316 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2022-05-07T15:52:26.316 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2022-05-07T15:52:26.317 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2022-05-07T15:52:26.317 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2022-05-07T15:52:26.317 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2022-05-07T15:52:26.317 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2022-05-07T15:52:26.318 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2022-05-07T15:52:26.318 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2022-05-07T15:52:26.318 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2022-05-07T15:52:26.318 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2022-05-07T15:52:26.319 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.319 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2022-05-07T15:52:26.319 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.319 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2022-05-07T15:52:26.320 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.320 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2022-05-07T15:52:26.320 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2022-05-07T15:52:26.320 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:52:26.321 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.321 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.321 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.321 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2022-05-07T15:52:26.322 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2022-05-07T15:52:26.322 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.322 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.322 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.323 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2022-05-07T15:52:26.323 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2022-05-07T15:52:26.323 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2022-05-07T15:52:26.323 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2022-05-07T15:52:26.324 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.324 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.324 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.324 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2022-05-07T15:52:26.325 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2022-05-07T15:52:26.325 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.325 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2022-05-07T15:52:26.325 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 41, 2022-05-07T15:52:26.326 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "c67bfd60-15c3-4eb2-a02f-223bbda3f2bb", 2022-05-07T15:52:26.326 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2022-05-07T15:50:37.990603+0000", 2022-05-07T15:52:26.326 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:51:52.548777+0000", 2022-05-07T15:52:26.326 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2022-05-07T15:51:51.126898+0000", 2022-05-07T15:52:26.327 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2022-05-07T15:51:52.269060+0000", 2022-05-07T15:52:26.327 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-05-07T15:52:26.327 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2022-05-07T15:52:26.327 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2022-05-07T15:52:26.328 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2022-05-07T15:52:26.328 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2022-05-07T15:52:26.328 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2022-05-07T15:52:26.328 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2022-05-07T15:52:26.329 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2022-05-07T15:52:26.329 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2022-05-07T15:52:26.329 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2022-05-07T15:52:26.329 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.330 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 9, 2022-05-07T15:52:26.330 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2022-05-07T15:52:26.330 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2022-05-07T15:52:26.330 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2022-05-07T15:52:26.331 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2022-05-07T15:52:26.331 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2022-05-07T15:52:26.331 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2022-05-07T15:52:26.331 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2022-05-07T15:52:26.332 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2022-05-07T15:52:26.332 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2022-05-07T15:52:26.332 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2022-05-07T15:52:26.332 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.332 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2022-05-07T15:52:26.333 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2022-05-07T15:52:26.333 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2022-05-07T15:51:02.550280+0000", 2022-05-07T15:52:26.333 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2022-05-07T15:52:26.333 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2022-05-07T15:52:26.334 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2022-05-07T15:52:26.334 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2022-05-07T15:52:26.334 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2022-05-07T15:52:26.334 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2022-05-07T15:52:26.334 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2022-05-07T15:52:26.335 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2022-05-07T15:52:26.335 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2022-05-07T15:52:26.335 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2022-05-07T15:52:26.335 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2022-05-07T15:52:26.336 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2022-05-07T15:52:26.336 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2022-05-07T15:52:26.336 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2022-05-07T15:52:26.336 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2022-05-07T15:52:26.337 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2022-05-07T15:52:26.337 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2022-05-07T15:52:26.337 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2022-05-07T15:52:26.337 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2022-05-07T15:52:26.337 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2022-05-07T15:52:26.338 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.338 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "29", 2022-05-07T15:52:26.338 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2022-05-07T15:52:26.338 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2022-05-07T15:52:26.339 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2022-05-07T15:52:26.339 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2022-05-07T15:52:26.339 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2022-05-07T15:52:26.340 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2022-05-07T15:52:26.340 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2022-05-07T15:52:26.340 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2022-05-07T15:52:26.340 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2022-05-07T15:52:26.340 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2022-05-07T15:52:26.341 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2022-05-07T15:52:26.341 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2022-05-07T15:52:26.341 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2022-05-07T15:52:26.341 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2022-05-07T15:52:26.342 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2022-05-07T15:52:26.342 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2022-05-07T15:52:26.342 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2022-05-07T15:52:26.342 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2022-05-07T15:52:26.343 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-05-07T15:52:26.343 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-05-07T15:52:26.343 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2022-05-07T15:52:26.343 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2022-05-07T15:52:26.344 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2022-05-07T15:52:26.344 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2022-05-07T15:52:26.344 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2022-05-07T15:52:26.345 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2022-05-07T15:52:26.345 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.345 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2022-05-07T15:52:26.345 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2022-05-07T15:52:26.345 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2022-05-07T15:52:26.346 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2022-05-07T15:52:26.346 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2022-05-07T15:52:26.346 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2022-05-07T15:52:26.346 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2022-05-07T15:52:26.347 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2022-05-07T15:52:26.347 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2022-05-07T15:52:26.347 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2022-05-07T15:52:26.347 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2022-05-07T15:52:26.348 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2022-05-07T15:52:26.348 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2022-05-07T15:52:26.348 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.348 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.349 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2022-05-07T15:52:26.349 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.349 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:52:26.349 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3", 2022-05-07T15:52:26.350 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2022-05-07T15:52:26.350 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:52:26.350 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:52:26.350 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:52:26.351 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2022-05-07T15:52:26.351 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2022-05-07T15:52:26.351 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 36, 2022-05-07T15:52:26.351 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 40, 2022-05-07T15:52:26.352 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2022-05-07T15:52:26.352 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:52:26.352 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:52:26.352 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.352 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.353 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.353 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2022-05-07T15:52:26.353 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.353 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.354 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.354 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.354 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2022-05-07T15:52:26.354 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.355 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.355 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.355 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.355 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:52:26.356 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.356 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.356 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.356 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2022-05-07T15:52:26.357 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.357 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.357 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.357 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.358 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2022-05-07T15:52:26.358 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.358 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.358 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.359 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.359 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:52:26.359 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.359 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.359 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.360 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2022-05-07T15:52:26.360 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.360 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.360 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.361 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.361 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2022-05-07T15:52:26.361 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.361 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.362 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.362 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.362 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:52:26.362 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.363 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.363 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.363 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2022-05-07T15:52:26.363 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.364 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.364 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.364 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.364 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2022-05-07T15:52:26.364 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19006 2022-05-07T15:52:26.365 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.365 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.365 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.366 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/19006", 2022-05-07T15:52:26.366 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/19006", 2022-05-07T15:52:26.366 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/19006", 2022-05-07T15:52:26.366 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/19006", 2022-05-07T15:52:26.366 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:52:26.367 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2022-05-07T15:52:26.367 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2022-05-07T15:52:26.367 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.368 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.368 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.368 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:52:26.368 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "56846436-68b9-4d7a-82ba-c1f770c44ac1", 2022-05-07T15:52:26.368 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2022-05-07T15:52:26.369 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:52:26.369 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:52:26.369 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:52:26.370 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2022-05-07T15:52:26.370 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2022-05-07T15:52:26.370 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2022-05-07T15:52:26.370 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 23, 2022-05-07T15:52:26.371 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2022-05-07T15:52:26.371 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:52:26.371 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:52:26.371 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.371 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.372 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.372 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2022-05-07T15:52:26.372 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.373 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.373 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.373 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.373 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2022-05-07T15:52:26.374 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.374 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.374 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.374 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.375 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:52:26.375 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.375 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.375 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.375 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2022-05-07T15:52:26.376 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.376 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.376 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.376 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.377 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2022-05-07T15:52:26.377 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.377 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.377 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.378 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.378 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:52:26.378 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.379 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.379 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.379 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2022-05-07T15:52:26.379 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.379 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.380 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.380 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.380 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2022-05-07T15:52:26.380 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.381 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.381 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.381 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.381 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:52:26.382 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.382 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.382 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.382 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2022-05-07T15:52:26.383 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.383 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.383 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.383 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.384 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2022-05-07T15:52:26.384 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 14628 2022-05-07T15:52:26.384 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.384 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.385 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.385 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/14628", 2022-05-07T15:52:26.385 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/14628", 2022-05-07T15:52:26.385 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/14628", 2022-05-07T15:52:26.385 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/14628", 2022-05-07T15:52:26.386 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:52:26.386 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2022-05-07T15:52:26.386 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.386 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.387 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.387 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:52:26.387 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "6c079b36-581d-4b23-8fed-4f24d6c517ec", 2022-05-07T15:52:26.387 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2022-05-07T15:52:26.388 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:52:26.388 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:52:26.388 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:52:26.388 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2022-05-07T15:52:26.389 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2022-05-07T15:52:26.389 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 38, 2022-05-07T15:52:26.389 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2022-05-07T15:52:26.389 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2022-05-07T15:52:26.390 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:52:26.390 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:52:26.390 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.390 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.391 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.391 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2022-05-07T15:52:26.391 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.391 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.392 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.392 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.392 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2022-05-07T15:52:26.392 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.393 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.393 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.393 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.393 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:52:26.393 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.394 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.394 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.394 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2022-05-07T15:52:26.394 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.395 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.395 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.395 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.395 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2022-05-07T15:52:26.396 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.396 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.396 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.396 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.397 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:52:26.397 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.397 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.397 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.398 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2022-05-07T15:52:26.398 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.398 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.398 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.399 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.399 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2022-05-07T15:52:26.399 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.399 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.400 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.400 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.400 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:52:26.400 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:52:26.400 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.401 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:52:26.401 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2022-05-07T15:52:26.401 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.401 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.402 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.402 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:52:26.402 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2022-05-07T15:52:26.402 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 19200 2022-05-07T15:52:26.403 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.403 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.403 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.403 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/19200", 2022-05-07T15:52:26.403 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/19200", 2022-05-07T15:52:26.404 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/19200", 2022-05-07T15:52:26.404 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/19200", 2022-05-07T15:52:26.404 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:52:26.404 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2022-05-07T15:52:26.405 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2022-05-07T15:52:26.405 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.405 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.405 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.406 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2022-05-07T15:52:26.406 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.406 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:52:26.406 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:51:44.762843+0000", 2022-05-07T15:52:26.407 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:52:26.407 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:52:26.407 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:52:26.407 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:52:26.407 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:50:45.297944+0000", 2022-05-07T15:52:26.408 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:52:26.408 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.408 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.408 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:52:26.409 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:51:45.244232+0000", 2022-05-07T15:52:26.409 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:52:26.409 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:52:26.409 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:52:26.410 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:52:26.410 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:50:52.670171+0000", 2022-05-07T15:52:26.410 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:52:26.410 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.410 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.411 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:52:26.411 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:51:45.725997+0000", 2022-05-07T15:52:26.411 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:52:26.411 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:52:26.412 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:52:26.412 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:52:26.412 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:50:59.273447+0000", 2022-05-07T15:52:26.412 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:52:26.413 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.413 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.413 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2022-05-07T15:52:26.413 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2022-05-07T15:52:26.414 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2022-05-07T15:52:26.414 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2022-05-07T15:52:26.414 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2022-05-07T15:52:26.414 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2022-05-07T15:52:26.415 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2022-05-07T15:52:26.415 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2022-05-07T15:52:26.415 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2022-05-07T15:52:26.415 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2022-05-07T15:52:26.416 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2022-05-07T15:52:26.416 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.416 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.416 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2022-05-07T15:52:26.417 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2022-05-07T15:52:26.417 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2022-05-07T15:52:26.417 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2022-05-07T15:52:26.417 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2022-05-07T15:52:26.418 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.418 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2022-05-07T15:52:26.418 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.418 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:52:26.419 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:52:26.419 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/19006,v1:127.0.0.1:6805/19006]", 2022-05-07T15:52:26.419 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:52:26.419 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:52:26.420 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:52:26.420 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:52:26.420 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:52:26.420 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:52:26.421 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:52:26.421 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:52:26.421 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:52:26.422 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2022-05-07T15:52:26.422 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:52:26.422 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:52:26.422 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:52:26.423 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:52:26.423 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:52:26.423 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:52:26.423 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:52:26.424 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:52:26.424 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:52:26.424 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:52:26.424 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:52:26.424 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:52:26.425 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:52:26.425 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:52:26.425 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:52:26.425 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/19006,v1:127.0.0.1:6803/19006]", 2022-05-07T15:52:26.426 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:52:26.426 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/19006,v1:127.0.0.1:6809/19006]", 2022-05-07T15:52:26.426 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/19006,v1:127.0.0.1:6807/19006]", 2022-05-07T15:52:26.427 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:52:26.427 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:52:26.427 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:52:26.427 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:52:26.427 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:52:26.428 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:52:26.428 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:52:26.428 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:52:26.428 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:52:26.429 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2022-05-07T15:52:26.429 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:52:26.429 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:52:26.429 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:52:26.430 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.430 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.430 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:52:26.430 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:52:26.431 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/14628,v1:127.0.0.1:6813/14628]", 2022-05-07T15:52:26.431 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:52:26.431 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:52:26.432 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:52:26.432 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:52:26.432 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:52:26.433 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:52:26.433 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:52:26.433 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:52:26.433 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:52:26.434 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2022-05-07T15:52:26.434 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:52:26.434 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:52:26.434 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:52:26.434 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:52:26.435 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:52:26.435 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:52:26.435 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:52:26.435 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:52:26.436 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:52:26.436 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:52:26.436 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:52:26.436 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:52:26.437 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:52:26.437 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:52:26.437 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:52:26.437 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/14628,v1:127.0.0.1:6811/14628]", 2022-05-07T15:52:26.438 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:52:26.438 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/14628,v1:127.0.0.1:6817/14628]", 2022-05-07T15:52:26.438 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/14628,v1:127.0.0.1:6815/14628]", 2022-05-07T15:52:26.438 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:52:26.438 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:52:26.439 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:52:26.439 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:52:26.439 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:52:26.440 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:52:26.440 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:52:26.440 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:52:26.440 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:52:26.441 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2022-05-07T15:52:26.441 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:52:26.441 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:52:26.441 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:52:26.441 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.442 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.442 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:52:26.442 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:52:26.442 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/19200,v1:127.0.0.1:6813/19200]", 2022-05-07T15:52:26.443 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:52:26.443 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:52:26.443 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:52:26.443 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:52:26.444 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:52:26.444 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:52:26.444 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:52:26.444 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:52:26.445 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:52:26.445 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2022-05-07T15:52:26.445 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:52:26.445 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:52:26.445 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:52:26.446 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:52:26.446 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:52:26.446 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:52:26.446 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:52:26.447 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:52:26.447 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:52:26.447 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:52:26.447 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:52:26.448 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:52:26.448 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:52:26.448 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:52:26.448 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:52:26.448 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/19200,v1:127.0.0.1:6811/19200]", 2022-05-07T15:52:26.449 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:52:26.449 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/19200,v1:127.0.0.1:6817/19200]", 2022-05-07T15:52:26.449 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/19200,v1:127.0.0.1:6815/19200]", 2022-05-07T15:52:26.449 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:52:26.450 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:52:26.450 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:52:26.450 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:52:26.450 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:52:26.451 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:52:26.451 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:52:26.451 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:52:26.451 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:52:26.452 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2022-05-07T15:52:26.452 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:52:26.452 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:52:26.452 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:52:26.452 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.453 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.453 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_clean_epochs": { 2022-05-07T15:52:26.453 INFO:tasks.workunit.client.0.smithi059.stdout: "min_last_epoch_clean": 0, 2022-05-07T15:52:26.453 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": { 2022-05-07T15:52:26.454 INFO:tasks.workunit.client.0.smithi059.stdout: "per_pool": [] 2022-05-07T15:52:26.454 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.454 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_epochs": [ 2022-05-07T15:52:26.454 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.455 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:52:26.455 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 36 2022-05-07T15:52:26.455 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.455 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.456 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:52:26.456 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 10 2022-05-07T15:52:26.456 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.456 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.457 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:52:26.457 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 39 2022-05-07T15:52:26.457 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.457 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.457 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.458 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2022-05-07T15:52:26.458 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 41, 2022-05-07T15:52:26.458 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2022-05-07T15:52:26.458 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2022-05-07T15:52:26.459 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.459 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:52:26.459 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2022-05-07T15:52:26.459 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:52:26.460 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.460 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.460 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:52:26.461 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2022-05-07T15:52:26.461 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:52:26.461 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.461 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.462 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:52:26.462 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2022-05-07T15:52:26.462 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:52:26.462 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.463 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.463 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2022-05-07T15:52:26.463 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.463 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2022-05-07T15:52:26.464 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2022-05-07T15:52:26.464 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.464 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.464 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:52:26.464 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2022-05-07T15:52:26.465 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.465 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.465 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2022-05-07T15:52:26.465 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2022-05-07T15:52:26.466 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.466 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.466 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2022-05-07T15:52:26.466 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2022-05-07T15:52:26.467 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.467 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.467 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2022-05-07T15:52:26.467 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2022-05-07T15:52:26.468 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.468 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.468 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2022-05-07T15:52:26.468 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2022-05-07T15:52:26.469 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.469 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.469 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2022-05-07T15:52:26.469 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2022-05-07T15:52:26.470 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.470 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.470 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2022-05-07T15:52:26.470 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2022-05-07T15:52:26.471 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.471 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.471 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2022-05-07T15:52:26.471 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2022-05-07T15:52:26.472 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.472 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.472 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2022-05-07T15:52:26.472 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2022-05-07T15:52:26.473 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.473 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.473 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2022-05-07T15:52:26.474 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2022-05-07T15:52:26.474 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.474 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.474 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:52:26.475 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2022-05-07T15:52:26.475 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.475 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.475 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2022-05-07T15:52:26.476 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.476 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2022-05-07T15:52:26.476 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2022-05-07T15:52:26.476 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:52:26.477 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2022-05-07T15:52:26.477 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:52:26.477 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:52:26.477 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:52:26.478 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:52:26.478 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.478 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2022-05-07T15:52:26.478 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:52:26.479 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:52:26.479 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.479 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.479 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.480 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.480 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2022-05-07T15:52:26.480 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2022-05-07T15:52:26.481 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:52:26.481 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2022-05-07T15:52:26.481 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:52:26.481 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:52:26.482 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:52:26.482 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:52:26.482 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.482 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2022-05-07T15:52:26.483 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:52:26.483 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:52:26.483 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.483 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.484 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.484 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.484 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2022-05-07T15:52:26.484 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2022-05-07T15:52:26.484 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:52:26.485 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2022-05-07T15:52:26.485 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:52:26.485 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:52:26.485 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:52:26.486 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:52:26.486 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.486 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:52:26.486 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:52:26.487 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:52:26.487 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.487 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.487 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:52:26.488 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:52:26.488 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2022-05-07T15:52:26.488 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.488 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.489 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:52:26.489 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:52:26.489 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2022-05-07T15:52:26.489 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.490 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.490 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.490 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.490 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2022-05-07T15:52:26.491 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2022-05-07T15:52:26.491 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:52:26.491 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2022-05-07T15:52:26.491 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:52:26.491 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:52:26.492 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:52:26.492 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:52:26.492 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.492 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:52:26.493 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:52:26.493 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:52:26.493 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.493 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.493 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:52:26.494 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:52:26.494 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2022-05-07T15:52:26.494 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.494 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.495 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:52:26.495 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:52:26.495 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2022-05-07T15:52:26.495 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.496 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.496 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.496 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.496 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2022-05-07T15:52:26.497 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.497 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2022-05-07T15:52:26.497 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2022-05-07T15:52:26.497 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2022-05-07T15:52:26.498 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2022-05-07T15:52:26.498 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2022-05-07T15:52:26.498 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2022-05-07T15:52:26.498 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2022-05-07T15:52:26.499 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.499 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2022-05-07T15:52:26.499 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2022-05-07T15:52:26.499 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2022-05-07T15:52:26.499 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.500 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.500 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2022-05-07T15:52:26.500 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2022-05-07T15:52:26.500 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2022-05-07T15:52:26.501 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.501 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.501 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2022-05-07T15:52:26.501 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.502 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:52:26.502 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.502 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.502 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2022-05-07T15:52:26.502 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2022-05-07T15:52:26.503 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2022-05-07T15:52:26.503 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2022-05-07T15:52:26.503 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2022-05-07T15:52:26.503 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2022-05-07T15:52:26.504 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2022-05-07T15:52:26.504 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2022-05-07T15:52:26.504 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2022-05-07T15:52:26.504 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2022-05-07T15:52:26.505 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2022-05-07T15:52:26.505 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2022-05-07T15:52:26.505 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2022-05-07T15:52:26.505 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2022-05-07T15:52:26.506 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2022-05-07T15:52:26.506 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2022-05-07T15:52:26.506 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2022-05-07T15:52:26.506 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2022-05-07T15:52:26.506 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2022-05-07T15:52:26.507 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2022-05-07T15:52:26.507 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2022-05-07T15:52:26.507 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.507 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2022-05-07T15:52:26.508 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.508 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2022-05-07T15:52:26.508 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2022-05-07T15:52:26.508 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2022-05-07T15:52:26.509 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2022-05-07T15:52:26.509 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2022-05-07T15:52:26.509 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2022-05-07T15:52:26.509 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2022-05-07T15:52:26.509 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2022-05-07T15:52:26.510 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2022-05-07T15:52:26.510 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2022-05-07T15:52:26.510 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2022-05-07T15:52:26.510 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2022-05-07T15:52:26.511 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2022-05-07T15:52:26.511 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2022-05-07T15:52:26.511 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2022-05-07T15:52:26.511 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2022-05-07T15:52:26.512 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.512 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.512 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2022-05-07T15:52:26.512 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2022-05-07T15:52:26.513 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2022-05-07T15:52:26.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.513 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2022-05-07T15:52:26.513 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2022-05-07T15:52:26.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.514 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2022-05-07T15:52:26.514 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2022-05-07T15:52:26.514 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2022-05-07T15:52:26.514 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2022-05-07T15:52:26.515 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2022-05-07T15:52:26.515 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2022-05-07T15:52:26.515 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.515 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2022-05-07T15:52:26.516 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2022-05-07T15:52:26.516 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2022-05-07T15:52:26.516 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2022-05-07T15:52:26.516 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2022-05-07T15:52:26.517 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2022-05-07T15:52:26.517 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 61700, 2022-05-07T15:52:26.517 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2022-05-07T15:52:26.517 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2022-05-07T15:52:26.517 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2022-05-07T15:52:26.518 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2022-05-07T15:52:26.518 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2022-05-07T15:52:26.518 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2022-05-07T15:52:26.518 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2022-05-07T15:52:26.519 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2022-05-07T15:52:26.519 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2022-05-07T15:52:26.519 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2022-05-07T15:52:26.519 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2022-05-07T15:52:26.520 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2022-05-07T15:52:26.520 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2022-05-07T15:52:26.520 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2022-05-07T15:52:26.520 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2022-05-07T15:52:26.521 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2022-05-07T15:52:26.521 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2022-05-07T15:52:26.521 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2022-05-07T15:52:26.521 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2022-05-07T15:52:26.521 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2022-05-07T15:52:26.522 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2022-05-07T15:52:26.522 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2022-05-07T15:52:26.522 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2022-05-07T15:52:26.522 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2022-05-07T15:52:26.523 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2022-05-07T15:52:26.523 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2022-05-07T15:52:26.523 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2022-05-07T15:52:26.523 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2022-05-07T15:52:26.524 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2022-05-07T15:52:26.524 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2022-05-07T15:52:26.524 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2022-05-07T15:52:26.524 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2022-05-07T15:52:26.525 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2022-05-07T15:52:26.525 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2022-05-07T15:52:26.525 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2022-05-07T15:52:26.525 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2022-05-07T15:52:26.525 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2022-05-07T15:52:26.526 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2022-05-07T15:52:26.526 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2022-05-07T15:52:26.526 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.526 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2022-05-07T15:52:26.527 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2022-05-07T15:52:26.527 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2022-05-07T15:52:26.527 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2022-05-07T15:52:26.527 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2022-05-07T15:52:26.528 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2022-05-07T15:52:26.528 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:52:26.528 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:52:26.528 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:52:26.529 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2022-05-07T15:52:26.529 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2022-05-07T15:52:26.529 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.529 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2022-05-07T15:52:26.529 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2022-05-07T15:52:26.530 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2022-05-07T15:52:26.530 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2022-05-07T15:52:26.530 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2022-05-07T15:52:26.530 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.531 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2022-05-07T15:52:26.531 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2022-05-07T15:52:26.531 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2022-05-07T15:52:26.531 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2022-05-07T15:52:26.532 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2022-05-07T15:52:26.532 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2022-05-07T15:52:26.532 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2022-05-07T15:52:26.532 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2022-05-07T15:52:26.533 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179776, 2022-05-07T15:52:26.533 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33792, 2022-05-07T15:52:26.533 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2022-05-07T15:52:26.533 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2022-05-07T15:52:26.533 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393024, 2022-05-07T15:52:26.534 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2022-05-07T15:52:26.534 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2022-05-07T15:52:26.534 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866456576, 2022-05-07T15:52:26.534 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2022-05-07T15:52:26.535 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34603008, 2022-05-07T15:52:26.535 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 458476, 2022-05-07T15:52:26.535 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:52:26.535 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:52:26.536 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:52:26.536 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13708, 2022-05-07T15:52:26.536 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211764 2022-05-07T15:52:26.536 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.537 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2022-05-07T15:52:26.537 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2022-05-07T15:52:26.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2022-05-07T15:52:26.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 1, 2022-05-07T15:52:26.537 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2022-05-07T15:52:26.538 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2022-05-07T15:52:26.538 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2022-05-07T15:52:26.538 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.538 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2022-05-07T15:52:26.539 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2022-05-07T15:52:26.539 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2022-05-07T15:52:26.539 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2022-05-07T15:52:26.539 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2022-05-07T15:52:26.540 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.540 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2022-05-07T15:52:26.540 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2022-05-07T15:52:26.540 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.540 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2022-05-07T15:52:26.541 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2022-05-07T15:52:26.541 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2022-05-07T15:52:26.541 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2022-05-07T15:52:26.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2022-05-07T15:52:26.542 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2022-05-07T15:52:26.542 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2022-05-07T15:52:26.542 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2022-05-07T15:52:26.542 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2022-05-07T15:52:26.543 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179776, 2022-05-07T15:52:26.543 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33792, 2022-05-07T15:52:26.543 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2022-05-07T15:52:26.543 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2022-05-07T15:52:26.543 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393024, 2022-05-07T15:52:26.544 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2022-05-07T15:52:26.544 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2022-05-07T15:52:26.544 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866456576, 2022-05-07T15:52:26.544 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2022-05-07T15:52:26.545 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34603008, 2022-05-07T15:52:26.545 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 458476, 2022-05-07T15:52:26.545 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:52:26.545 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:52:26.546 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:52:26.546 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13708, 2022-05-07T15:52:26.546 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211764 2022-05-07T15:52:26.546 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.547 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2022-05-07T15:52:26.547 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2022-05-07T15:52:26.547 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2022-05-07T15:52:26.547 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 1, 2022-05-07T15:52:26.548 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2022-05-07T15:52:26.548 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2022-05-07T15:52:26.548 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2022-05-07T15:52:26.548 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.549 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2022-05-07T15:52:26.549 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2022-05-07T15:52:26.549 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2022-05-07T15:52:26.549 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2022-05-07T15:52:26.550 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2022-05-07T15:52:26.550 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.550 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2022-05-07T15:52:26.550 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2022-05-07T15:52:26.550 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.551 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.551 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2022-05-07T15:52:26.551 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.551 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2022-05-07T15:52:26.552 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2022-05-07T15:52:26.552 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2022-05-07T15:52:26.552 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 61700, 2022-05-07T15:52:26.552 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2022-05-07T15:52:26.553 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2022-05-07T15:52:26.553 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2022-05-07T15:52:26.553 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2022-05-07T15:52:26.553 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2022-05-07T15:52:26.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2022-05-07T15:52:26.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2022-05-07T15:52:26.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2022-05-07T15:52:26.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2022-05-07T15:52:26.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2022-05-07T15:52:26.555 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2022-05-07T15:52:26.555 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2022-05-07T15:52:26.555 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2022-05-07T15:52:26.555 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2022-05-07T15:52:26.556 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2022-05-07T15:52:26.556 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2022-05-07T15:52:26.556 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2022-05-07T15:52:26.556 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2022-05-07T15:52:26.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2022-05-07T15:52:26.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2022-05-07T15:52:26.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2022-05-07T15:52:26.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2022-05-07T15:52:26.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2022-05-07T15:52:26.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2022-05-07T15:52:26.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2022-05-07T15:52:26.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2022-05-07T15:52:26.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2022-05-07T15:52:26.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2022-05-07T15:52:26.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2022-05-07T15:52:26.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2022-05-07T15:52:26.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2022-05-07T15:52:26.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2022-05-07T15:52:26.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2022-05-07T15:52:26.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2022-05-07T15:52:26.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2022-05-07T15:52:26.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2022-05-07T15:52:26.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2022-05-07T15:52:26.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2022-05-07T15:52:26.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2022-05-07T15:52:26.562 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.562 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2022-05-07T15:52:26.562 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2022-05-07T15:52:26.562 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2022-05-07T15:52:26.563 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2022-05-07T15:52:26.563 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19660800, 2022-05-07T15:52:26.563 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 185100, 2022-05-07T15:52:26.563 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:52:26.564 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:52:26.564 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:52:26.564 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2022-05-07T15:52:26.564 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2022-05-07T15:52:26.564 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.565 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2022-05-07T15:52:26.565 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2022-05-07T15:52:26.565 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2022-05-07T15:52:26.565 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2022-05-07T15:52:26.566 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2022-05-07T15:52:26.566 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.566 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.566 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2022-05-07T15:52:26.566 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.567 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:52:26.567 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 154618822666 2022-05-07T15:52:26.567 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.567 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.568 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:52:26.568 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672977 2022-05-07T15:52:26.568 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.568 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.569 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:52:26.569 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 163208757257 2022-05-07T15:52:26.569 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.569 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.569 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2022-05-07T15:52:26.570 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.570 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2022-05-07T15:52:26.570 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2022-05-07T15:52:26.570 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.571 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.571 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2022-05-07T15:52:26.571 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.571 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:52:26.572 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2022-05-07T15:52:26.572 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2022-05-07T15:52:26.572 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:52:26.572 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.573 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.573 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:52:26.573 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2022-05-07T15:52:26.573 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2022-05-07T15:52:26.574 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:52:26.574 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.574 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.574 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:52:26.575 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2022-05-07T15:52:26.575 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2022-05-07T15:52:26.575 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:52:26.575 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.576 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.576 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2022-05-07T15:52:26.576 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:52:26.576 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2022-05-07T15:52:26.576 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2022-05-07T15:52:26.577 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.577 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:52:26.577 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2022-05-07T15:52:26.577 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 4, 2022-05-07T15:52:26.578 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:51:54.000370+0000", 2022-05-07T15:52:26.578 INFO:tasks.workunit.client.0.smithi059.stdout: "services": { 2022-05-07T15:52:26.578 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": { 2022-05-07T15:52:26.578 INFO:tasks.workunit.client.0.smithi059.stdout: "daemons": { 2022-05-07T15:52:26.579 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": "", 2022-05-07T15:52:26.579 INFO:tasks.workunit.client.0.smithi059.stdout: "0": { 2022-05-07T15:52:26.579 INFO:tasks.workunit.client.0.smithi059.stdout: "start_epoch": 0, 2022-05-07T15:52:26.579 INFO:tasks.workunit.client.0.smithi059.stdout: "start_stamp": "0.000000", 2022-05-07T15:52:26.579 INFO:tasks.workunit.client.0.smithi059.stdout: "gid": 0, 2022-05-07T15:52:26.580 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "(unrecognized address family 0)/0", 2022-05-07T15:52:26.580 INFO:tasks.workunit.client.0.smithi059.stdout: "metadata": {}, 2022-05-07T15:52:26.580 INFO:tasks.workunit.client.0.smithi059.stdout: "task_status": {} 2022-05-07T15:52:26.580 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.581 INFO:tasks.workunit.client.0.smithi059.stdout: "2": { 2022-05-07T15:52:26.581 INFO:tasks.workunit.client.0.smithi059.stdout: "start_epoch": 0, 2022-05-07T15:52:26.581 INFO:tasks.workunit.client.0.smithi059.stdout: "start_stamp": "0.000000", 2022-05-07T15:52:26.581 INFO:tasks.workunit.client.0.smithi059.stdout: "gid": 0, 2022-05-07T15:52:26.581 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "(unrecognized address family 0)/0", 2022-05-07T15:52:26.582 INFO:tasks.workunit.client.0.smithi059.stdout: "metadata": {}, 2022-05-07T15:52:26.582 INFO:tasks.workunit.client.0.smithi059.stdout: "task_status": {} 2022-05-07T15:52:26.582 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.582 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.583 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.583 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.583 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:52:26.583 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2022-05-07T15:52:26.584 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2022-05-07T15:52:26.584 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 355, 2022-05-07T15:52:26.584 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2022-05-07T15:52:26.584 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2022-05-07T15:52:26.584 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:52:26.585 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:52:26.586 INFO:tasks.workunit.client.0.smithi059.stderr:report 1405454384 2022-05-07T15:52:26.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2022-05-07T15:52:26.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2022-05-07T15:52:26.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2022-05-07T15:52:26.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2022-05-07T15:52:26.588 INFO:tasks.workunit.client.0.smithi059.stdout:writing non-divergent object existing_4 2022-05-07T15:52:26.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:26.614 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:26.614 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2022-05-07T15:51:52.883784+0000 29'100 41:119 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:26.615 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:26.615 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:26.616 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:26.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2022-05-07T15:52:26.728 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery 2022-05-07T15:52:26.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2022-05-07T15:52:26.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:27.023 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:27.024 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2022-05-07T15:51:52.883784+0000 29'100 41:119 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:27.024 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:27.024 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:27.026 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:27.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2022-05-07T15:52:27.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2022-05-07T15:52:27.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-05-07T15:52:27.035 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-05-07T15:52:27.036 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:52:27.036 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:27.036 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:27.036 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.0.asok 2022-05-07T15:52:27.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2022-05-07T15:52:27.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.0.asok set_recovery_delay 100000 2022-05-07T15:52:27.111 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-05-07T15:52:27.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2022-05-07T15:52:27.119 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2022-05-07T15:52:27.119 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-05-07T15:52:27.120 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-05-07T15:52:27.120 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:52:27.120 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:27.121 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:27.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.2.asok 2022-05-07T15:52:27.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2022-05-07T15:52:27.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.2.asok set_recovery_delay 100000 2022-05-07T15:52:27.203 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-05-07T15:52:27.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2022-05-07T15:52:27.211 INFO:tasks.workunit.client.0.smithi059.stdout:revive divergent 1 2022-05-07T15:52:27.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:27.516 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:27.516 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2022-05-07T15:51:52.883784+0000 29'100 41:119 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:27.516 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:27.517 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:27.518 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:27.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2022-05-07T15:52:28.226 INFO:tasks.workunit.client.0.smithi059.stderr:noup is set 2022-05-07T15:52:28.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2022-05-07T15:52:28.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:52:28.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:52:28.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-05-07T15:52:28.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:52:28.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:52:28.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:52:28.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:52:28.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:52:28.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:52:28.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:52:28.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:52:28.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:52:28.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:52:28.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:52:28.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:52:28.244 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:52:28.244 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:52:28.246 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:52:28.247 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:28.247 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:28.247 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:52:28.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:52:28.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:52:28.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:52:28.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:52:28.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:52:28.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:52:28.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:52:28.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:52:28.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:52:28.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:52:28.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-05-07T15:52:28.250 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:52:28.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:52:28.252 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-05-07T15:52:28.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-05-07T15:52:28.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:52:28.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:52:28.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:52:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:28.283+0000 7f35b78aed80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:52:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:28.283+0000 7f35b78aed80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:52:28.286 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:28.287+0000 7f35b78aed80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:52:28.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2022-05-07T15:52:28.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2022-05-07T15:52:28.820 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:28.819+0000 7f35b78aed80 -1 Falling back to public interface 2022-05-07T15:52:29.715 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:29.711+0000 7f35b78aed80 -1 osd.1 29 log_to_monitors {default=true} 2022-05-07T15:52:33.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2022-05-07T15:52:33.574 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery divergent 2022-05-07T15:52:33.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:33.894 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:33.895 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61856 0 0 17 17 active+undersized+degraded 2022-05-07T15:51:52.883784+0000 41'101 42:122 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:33.895 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:33.895 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:33.897 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:33.906 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2022-05-07T15:52:33.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-05-07T15:52:33.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-05-07T15:52:33.908 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:52:33.908 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:33.908 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:33.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.1.asok 2022-05-07T15:52:33.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2022-05-07T15:52:33.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.1.asok set_recovery_delay 100000 2022-05-07T15:52:34.000 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-05-07T15:52:34.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2022-05-07T15:52:34.624 INFO:tasks.workunit.client.0.smithi059.stderr:noup is unset 2022-05-07T15:52:34.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2022-05-07T15:52:34.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:52:34.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:52:34.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:52:34.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:52:34.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:52:34.637 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:52:34.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:52:34.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:52:34.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:52:35.022 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/19006,v1:127.0.0.1:6803/19006] [v2:127.0.0.1:6804/19006,v1:127.0.0.1:6805/19006] exists,up cd9e0083-3741-4c3f-9e67-4cf45e0ccbc3 2022-05-07T15:52:35.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:52:35.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:52:35.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:52:35.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2022-05-07T15:52:35.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:52:35.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:52:35.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:52:35.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:52:35.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:52:35.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:52:35.027 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:52:35.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:52:35.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:52:35.363 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/20627,v1:127.0.0.1:6819/20627] [v2:127.0.0.1:6820/20627,v1:127.0.0.1:6821/20627] exists,up 56846436-68b9-4d7a-82ba-c1f770c44ac1 2022-05-07T15:52:35.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:52:35.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:52:35.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:52:35.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2022-05-07T15:52:35.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:52:35.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:52:35.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:52:35.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:52:35.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:52:35.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:52:35.368 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:52:35.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:52:35.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:52:35.689 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/19200,v1:127.0.0.1:6811/19200] [v2:127.0.0.1:6812/19200,v1:127.0.0.1:6813/19200] exists,up 6c079b36-581d-4b23-8fed-4f24d6c517ec 2022-05-07T15:52:35.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:52:35.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:52:35.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:52:35.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:36.007 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:36.007 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2022-05-07T15:52:34.741018+0000 29'101 46:123 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:36.008 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:36.008 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:36.010 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:36.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2022-05-07T15:52:36.019 INFO:tasks.workunit.client.0.smithi059.stdout:wait for peering 2022-05-07T15:52:36.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:36.331 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:36.332 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2022-05-07T15:52:34.741018+0000 29'101 46:123 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:36.332 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:36.332 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:36.334 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:36.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2022-05-07T15:52:36.508 INFO:tasks.workunit.client.0.smithi059.stdout:killing divergent 1 2022-05-07T15:52:36.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2022-05-07T15:52:36.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:36.827 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:36.828 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2022-05-07T15:52:34.741018+0000 29'101 46:123 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:36.828 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:36.828 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:36.830 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:36.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2022-05-07T15:52:36.839 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:52:36.840 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:52:36.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:52:36.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:52:36.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:52:36.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:52:36.952 INFO:tasks.workunit.client.0.smithi059.stdout:reviving divergent 1 2022-05-07T15:52:36.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2022-05-07T15:52:36.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:37.273 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:37.273 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2022-05-07T15:52:34.741018+0000 29'101 46:123 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:37.274 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:37.274 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:37.275 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:37.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2022-05-07T15:52:37.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:52:37.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:52:37.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-05-07T15:52:37.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:52:37.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:52:37.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:52:37.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:52:37.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:52:37.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:52:37.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:52:37.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:52:37.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:52:37.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:52:37.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:52:37.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:52:37.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:52:37.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:52:37.293 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:52:37.293 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:37.293 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:37.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:52:37.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:52:37.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:52:37.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:52:37.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:52:37.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:52:37.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:52:37.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:52:37.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:52:37.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:52:37.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:52:37.297 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:52:37.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-05-07T15:52:37.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:52:37.298 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-05-07T15:52:37.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-05-07T15:52:37.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:52:37.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:52:37.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:52:37.324 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:37.323+0000 7f1a70195d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:52:37.326 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:37.327+0000 7f1a70195d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:52:37.329 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:37.327+0000 7f1a70195d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:52:37.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2022-05-07T15:52:37.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:52:37.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:52:37.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:52:37.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:52:37.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:52:37.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:52:37.629 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:52:37.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:52:37.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:52:37.860 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:37.859+0000 7f1a70195d80 -1 Falling back to public interface 2022-05-07T15:52:37.960 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/20627,v1:127.0.0.1:6819/20627] [v2:127.0.0.1:6820/20627,v1:127.0.0.1:6821/20627] exists,up 56846436-68b9-4d7a-82ba-c1f770c44ac1 2022-05-07T15:52:37.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:52:37.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:52:37.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:52:37.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2022-05-07T15:52:38.807 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:52:38.807+0000 7f1a70195d80 -1 osd.1 47 log_to_monitors {default=true} 2022-05-07T15:52:57.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2022-05-07T15:52:57.963 INFO:tasks.workunit.client.0.smithi059.stdout:allowing recovery 2022-05-07T15:52:57.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:58.313 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:58.314 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 62473 0 0 18 18 active+clean 2022-05-07T15:52:40.495976+0000 47'102 52:149 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:58.314 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:58.314 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:58.316 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:58.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2022-05-07T15:52:58.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2022-05-07T15:52:58.464 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-05-07T15:52:58.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2022-05-07T15:52:58.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2022-05-07T15:52:58.618 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-05-07T15:52:58.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2022-05-07T15:52:58.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2022-05-07T15:52:58.767 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-05-07T15:52:58.775 INFO:tasks.workunit.client.0.smithi059.stdout:reading divergent objects 2022-05-07T15:52:58.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2022-05-07T15:52:58.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2022-05-07T15:52:59.092 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:52:59.093 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 62473 0 0 18 18 active+clean 2022-05-07T15:52:40.495976+0000 47'102 52:149 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:51:02.600501+0000 0'0 2022-05-07T15:51:02.600501+0000 0 2022-05-07T15:52:59.093 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:52:59.093 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:52:59.095 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:52:59.105 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2022-05-07T15:52:59.106 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2022-05-07T15:52:59.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:52:59.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2022-05-07T15:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2022-05-07T15:52:59.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:52:59.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2022-05-07T15:52:59.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:52:59.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2022-05-07T15:52:59.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2022-05-07T15:52:59.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2022-05-07T15:52:59.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2022-05-07T15:52:59.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4530.0:1 2022-05-07T15:51:33.482088+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2022-05-07T15:52:59.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4530.0:1 2022-05-07T15:51:33.482088+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2022-05-07T15:52:59.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-05-07T15:52:59.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2022-05-07T15:52:59.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2022-05-07T15:52:59.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2022-05-07T15:52:59.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2022-05-07T15:52:59.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4530.0:1 2022-05-07T15:51:33.482088+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2022-05-07T15:52:59.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:52:35.179+0000 7f359d5aa700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2022-05-07T15:52:59.294 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2022-05-07T15:52:59.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2022-05-07T15:52:59.301 INFO:tasks.workunit.client.0.smithi059.stdout:success 2022-05-07T15:52:59.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2022-05-07T15:52:59.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2022-05-07T15:52:59.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2022-05-07T15:52:59.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2022-05-07T15:52:59.773 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' does not exist 2022-05-07T15:52:59.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2022-05-07T15:52:59.782 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:52:59.783 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:52:59.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:52:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:52:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:52:59.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:52:59.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2022-05-07T15:52:59.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-05-07T15:52:59.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:52:59.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-05-07T15:52:59.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:52:59.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:52:59.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:52:59.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:52:59.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:52:59.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:52:59.921 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:52:59.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:52:59.923 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:52:59.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:52:59.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:52:59.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:52:59.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:52:59.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:52:59.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:52:59.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:52:59.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:52:59.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:52:59.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-05-07T15:52:59.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:52:59.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:59.946 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:59.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13828 2022-05-07T15:52:59.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:52:59.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:52:59.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2022-05-07T15:52:59.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2022-05-07T15:52:59.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2022-05-07T15:52:59.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2022-05-07T15:52:59.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-05-07T15:52:59.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:52:59.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-05-07T15:52:59.950 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:52:59.951 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:52:59.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:52:59.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:52:59.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:52:59.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:52:59.952 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:52:59.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:52:59.954 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:52:59.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:52:59.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:52:59.957 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:52:59.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:52:59.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:52:59.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:52:59.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:52:59.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:52:59.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:52:59.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-05-07T15:52:59.964 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:52:59.964 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:59.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:59.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13828 2022-05-07T15:52:59.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:52:59.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:52:59.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2022-05-07T15:52:59.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-05-07T15:52:59.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:52:59.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:52:59.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.13828 2022-05-07T15:52:59.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-05-07T15:52:59.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2022-05-07T15:52:59.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-05-07T15:52:59.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2022-05-07T15:52:59.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2022-05-07T15:52:59.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2022-05-07T15:52:59.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2022-05-07T15:52:59.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2022-05-07T15:52:59.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2022-05-07T15:52:59.974 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2022-05-07T15:52:59.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2022-05-07T15:52:59.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2022-05-07T15:52:59.977 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:52:59.977 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:52:59.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2022-05-07T15:52:59.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2022-05-07T15:52:59.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-05-07T15:52:59.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-05-07T15:52:59.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-05-07T15:52:59.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2022-05-07T15:52:59.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2022-05-07T15:53:00.214 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-05-07T15:53:00.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:53:00.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:53:00.216 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:53:00.216 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:00.216 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:00.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:00.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-05-07T15:53:00.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-05-07T15:53:00.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-05-07T15:53:00.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:53:00.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:53:00.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-05-07T15:53:00.387 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-05-07T15:53:00.387 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:53:00.387 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:53:00.387 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:53:00.388 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:53:00.388 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:00.388 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:00.389 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-mon.a.asok 2022-05-07T15:53:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:53:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.13828/ceph-mon.a.asok config get fsid 2022-05-07T15:53:00.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-05-07T15:53:00.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:53:00.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:53:00.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-05-07T15:53:00.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-05-07T15:53:00.479 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:53:00.480 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:53:00.480 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:53:00.481 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:53:00.481 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:00.481 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:00.481 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-mon.a.asok 2022-05-07T15:53:00.482 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:53:00.482 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.13828/ceph-mon.a.asok config get mon_host 2022-05-07T15:53:00.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2022-05-07T15:53:00.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2022-05-07T15:53:00.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-05-07T15:53:00.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-05-07T15:53:00.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-05-07T15:53:00.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2022-05-07T15:53:00.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-05-07T15:53:01.454 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-05-07T15:53:01.455 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:53:01.455 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:53:01.455 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:53:01.455 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:01.456 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:01.456 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:01.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:53:01.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:53:01.495 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2022-05-07T15:53:01.495 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2022-05-07T15:53:01.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:53:01.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:53:01.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-05-07T15:53:01.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:53:01.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2022-05-07T15:53:01.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:53:01.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:53:01.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:53:01.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:53:01.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-05-07T15:53:01.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-05-07T15:53:01.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:53:01.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:53:01.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:53:01.499 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:53:01.499 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:53:01.500 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:53:01.500 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:53:01.500 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:01.500 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:01.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:01.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:01.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:53:01.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:53:01.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:53:01.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:53:01.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:53:01.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:53:01.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:53:01.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:53:01.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:53:01.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:53:01.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2022-05-07T15:53:01.504 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:53:01.505 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 702a02ae-db92-4105-8037-f4f69248fb71 2022-05-07T15:53:01.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=702a02ae-db92-4105-8037-f4f69248fb71 2022-05-07T15:53:01.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 702a02ae-db92-4105-8037-f4f69248fb71' 2022-05-07T15:53:01.506 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:53:01.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDdlXZiV/53HxAASDDsF+IuQF1Rk4r2BTgI+w== 2022-05-07T15:53:01.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDdlXZiV/53HxAASDDsF+IuQF1Rk4r2BTgI+w=="}' 2022-05-07T15:53:01.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 702a02ae-db92-4105-8037-f4f69248fb71 -i td/divergent-priors/0/new.json 2022-05-07T15:53:01.835 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:53:01.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2022-05-07T15:53:01.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdlXZiV/53HxAASDDsF+IuQF1Rk4r2BTgI+w== --osd-uuid 702a02ae-db92-4105-8037-f4f69248fb71 2022-05-07T15:53:01.883 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.883+0000 7ff74504fd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:01.887 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.887+0000 7ff74504fd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:01.887 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.887+0000 7ff74504fd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:01.888 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.887+0000 7ff74504fd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:53:01.888 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.887+0000 7ff74504fd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:53:01.888 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.887+0000 7ff74504fd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:53:01.933 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:01.935+0000 7ff74504fd80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2022-05-07T15:53:02.343 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:02.343+0000 7ff74504fd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:53:05.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2022-05-07T15:53:05.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:53:05.034 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2022-05-07T15:53:05.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-05-07T15:53:05.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:53:05.468 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2022-05-07T15:53:05.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-05-07T15:53:05.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:53:05.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:53:05.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:53:05.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:53:05.507 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:05.507+0000 7fbea9eb8d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:05.510 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:05.511+0000 7fbea9eb8d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:05.511 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:05.511+0000 7fbea9eb8d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:05.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-05-07T15:53:05.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:53:05.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:53:05.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:53:05.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:53:05.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:05.810 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:53:05.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:53:05.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:05.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:53:06.043 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:06.043+0000 7fbea9eb8d80 -1 Falling back to public interface 2022-05-07T15:53:06.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:53:06.872 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:06.871+0000 7fbea9eb8d80 -1 osd.0 0 log_to_monitors {default=true} 2022-05-07T15:53:07.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:53:07.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:07.140 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:53:07.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:53:07.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:07.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:53:07.552 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:53:08.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:53:08.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:08.554 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:53:08.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:53:08.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:08.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:53:08.896 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/21902,v1:127.0.0.1:6803/21902] [v2:127.0.0.1:6804/21902,v1:127.0.0.1:6805/21902] exists,up 702a02ae-db92-4105-8037-f4f69248fb71 2022-05-07T15:53:08.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:53:08.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:53:08.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:53:08.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2022-05-07T15:53:08.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2022-05-07T15:53:08.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:53:08.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:53:08.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-05-07T15:53:08.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:53:08.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:53:08.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:53:08.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:53:08.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:53:08.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:53:08.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:53:08.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:53:08.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:53:08.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:53:08.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:53:08.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:53:08.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:53:08.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:53:08.908 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:53:08.908 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:08.909 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:08.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:08.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:08.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:53:08.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:53:08.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:53:08.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:53:08.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:53:08.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:53:08.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:53:08.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:53:08.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:53:08.912 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:53:08.912 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:53:08.913 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:53:08.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:53:08.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 389e3986-a3c9-4ea7-b566-1125c8d4e23a' 2022-05-07T15:53:08.914 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:53:08.914 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:53:08.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDklXZiV1bBNxAAwLrtKwRMXcoZ1gbmmHKJbw== 2022-05-07T15:53:08.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDklXZiV1bBNxAAwLrtKwRMXcoZ1gbmmHKJbw=="}' 2022-05-07T15:53:08.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 389e3986-a3c9-4ea7-b566-1125c8d4e23a -i td/divergent-priors/1/new.json 2022-05-07T15:53:09.397 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:53:09.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2022-05-07T15:53:09.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDklXZiV1bBNxAAwLrtKwRMXcoZ1gbmmHKJbw== --osd-uuid 389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:53:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.451+0000 7f3310ad0d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:09.456 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.455+0000 7f3310ad0d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:09.457 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.459+0000 7f3310ad0d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:09.458 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.459+0000 7f3310ad0d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:53:09.458 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.459+0000 7f3310ad0d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:53:09.458 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.459+0000 7f3310ad0d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:53:09.502 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.503+0000 7f3310ad0d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2022-05-07T15:53:09.929 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:09.931+0000 7f3310ad0d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:53:12.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2022-05-07T15:53:12.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:53:12.694 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2022-05-07T15:53:12.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-05-07T15:53:12.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:53:13.120 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:53:13.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-05-07T15:53:13.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:53:13.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:53:13.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:53:13.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:53:13.156 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:13.155+0000 7f6c15e3ad80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:13.159 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:13.159+0000 7f6c15e3ad80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:13.160 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:13.159+0000 7f6c15e3ad80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:13.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-05-07T15:53:13.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:53:13.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:53:13.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:53:13.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:53:13.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:13.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:53:13.476 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:53:13.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:13.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:53:13.711 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:13.711+0000 7f6c15e3ad80 -1 Falling back to public interface 2022-05-07T15:53:13.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:53:14.499 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:14.499+0000 7f6c15e3ad80 -1 osd.1 0 log_to_monitors {default=true} 2022-05-07T15:53:14.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:53:14.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:14.809 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:53:14.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:53:14.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:14.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:53:15.206 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/22285,v1:127.0.0.1:6811/22285] [v2:127.0.0.1:6812/22285,v1:127.0.0.1:6813/22285] exists,up 389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:53:15.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:53:15.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:53:15.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:53:15.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2022-05-07T15:53:15.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2022-05-07T15:53:15.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:53:15.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:53:15.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-05-07T15:53:15.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:53:15.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2022-05-07T15:53:15.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:53:15.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:53:15.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:53:15.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:53:15.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-05-07T15:53:15.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-05-07T15:53:15.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:53:15.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:53:15.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:53:15.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:53:15.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:53:15.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:53:15.214 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:53:15.214 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:15.214 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:15.216 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:15.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:53:15.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:53:15.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:53:15.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:53:15.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:53:15.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:53:15.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:53:15.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:53:15.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:53:15.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:53:15.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:53:15.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2022-05-07T15:53:15.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:53:15.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b47503be-d6f2-4cab-a385-68acebbdfb7e 2022-05-07T15:53:15.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b47503be-d6f2-4cab-a385-68acebbdfb7e' 2022-05-07T15:53:15.220 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 b47503be-d6f2-4cab-a385-68acebbdfb7e 2022-05-07T15:53:15.221 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:53:15.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDrlXZiaaVGDhAAChdkfYyPLhaSkNKW3qKpBQ== 2022-05-07T15:53:15.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDrlXZiaaVGDhAAChdkfYyPLhaSkNKW3qKpBQ=="}' 2022-05-07T15:53:15.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b47503be-d6f2-4cab-a385-68acebbdfb7e -i td/divergent-priors/2/new.json 2022-05-07T15:53:15.796 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:53:15.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2022-05-07T15:53:15.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDrlXZiaaVGDhAAChdkfYyPLhaSkNKW3qKpBQ== --osd-uuid b47503be-d6f2-4cab-a385-68acebbdfb7e 2022-05-07T15:53:15.848 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.847+0000 7f682c4bbd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:15.851 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.851+0000 7f682c4bbd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:15.853 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.855+0000 7f682c4bbd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:15.854 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.855+0000 7f682c4bbd80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:53:15.855 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.855+0000 7f682c4bbd80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:53:15.855 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.855+0000 7f682c4bbd80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:53:15.935 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:15.935+0000 7f682c4bbd80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2022-05-07T15:53:16.278 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:16.279+0000 7f682c4bbd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:53:19.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2022-05-07T15:53:19.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:53:19.076 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2022-05-07T15:53:19.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-05-07T15:53:19.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:53:19.497 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2022-05-07T15:53:19.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-05-07T15:53:19.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:53:19.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:53:19.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:53:19.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:53:19.535 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:19.535+0000 7f097aca6d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:19.537 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:19.539+0000 7f097aca6d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:19.538 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:19.539+0000 7f097aca6d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:53:19.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-05-07T15:53:19.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:53:19.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:53:19.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:53:19.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:53:19.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:19.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:53:19.834 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:53:19.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:19.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:53:20.071 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:20.071+0000 7f097aca6d80 -1 Falling back to public interface 2022-05-07T15:53:20.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:53:20.891 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:53:20.891+0000 7f097aca6d80 -1 osd.2 0 log_to_monitors {default=true} 2022-05-07T15:53:21.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:53:21.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:21.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:53:21.179 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:53:21.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:21.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:53:21.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:53:22.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:53:22.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:53:22.527 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:53:22.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:53:22.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:53:22.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:53:22.870 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/22640,v1:127.0.0.1:6819/22640] [v2:127.0.0.1:6820/22640,v1:127.0.0.1:6821/22640] exists,up b47503be-d6f2-4cab-a385-68acebbdfb7e 2022-05-07T15:53:22.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:53:22.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:53:22.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:53:22.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2022-05-07T15:53:23.568 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2022-05-07T15:53:23.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2022-05-07T15:53:24.329 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2022-05-07T15:53:24.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2022-05-07T15:53:24.988 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2022-05-07T15:53:25.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2022-05-07T15:53:25.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2022-05-07T15:53:25.570 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2022-05-07T15:53:25.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-05-07T15:53:26.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2022-05-07T15:53:27.239 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 3 2022-05-07T15:53:27.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2022-05-07T15:53:27.882 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2022-05-07T15:53:27.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2022-05-07T15:53:27.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:53:27.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:53:28.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:53:28.246 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:53:28.246 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:53:28.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:53:28.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:28.247 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:53:28.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2022-05-07T15:53:28.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2022-05-07T15:53:28.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2022-05-07T15:53:28.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:28.405 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:53:28.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2022-05-07T15:53:28.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2022-05-07T15:53:28.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2022-05-07T15:53:28.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:28.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:53:28.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509445 2022-05-07T15:53:28.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509445 2022-05-07T15:53:28.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509445' 2022-05-07T15:53:28.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:28.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2022-05-07T15:53:28.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:28.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:53:28.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2022-05-07T15:53:28.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:28.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2022-05-07T15:53:28.754 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836487 2022-05-07T15:53:28.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2022-05-07T15:53:28.755 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:53:29.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2022-05-07T15:53:29.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:29.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2022-05-07T15:53:29.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:29.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:53:29.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2022-05-07T15:53:29.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:29.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2022-05-07T15:53:29.098 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2022-05-07T15:53:29.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2022-05-07T15:53:29.098 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:53:29.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2022-05-07T15:53:29.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:29.432 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509445 2022-05-07T15:53:29.432 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:29.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:53:29.435 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509445 2022-05-07T15:53:29.435 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:29.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509445 2022-05-07T15:53:29.437 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509445 2022-05-07T15:53:29.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2022-05-07T15:53:29.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:53:29.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509443 -lt 64424509445 2022-05-07T15:53:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:53:30.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:53:30.811 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:53:31.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509445 -lt 64424509445 2022-05-07T15:53:31.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2022-05-07T15:53:31.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:53:31.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:53:31.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:53:31.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:53:31.158 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2022-05-07T15:53:31.158 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:53:31.159 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:53:31.159 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:53:31.159 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:53:31.159 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:53:31.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:53:31.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:53:31.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:53:31.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:53:31.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:53:31.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:53:31.742 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:53:31.742 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:53:31.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:53:31.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:31.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:53:31.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2022-05-07T15:53:31.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2022-05-07T15:53:31.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2022-05-07T15:53:31.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:31.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:53:32.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2022-05-07T15:53:32.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2022-05-07T15:53:32.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2022-05-07T15:53:32.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:32.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:53:32.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509447 2022-05-07T15:53:32.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509447 2022-05-07T15:53:32.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2022-05-07T15:53:32.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:32.194 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2022-05-07T15:53:32.194 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:32.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:53:32.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2022-05-07T15:53:32.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:32.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2022-05-07T15:53:32.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2022-05-07T15:53:32.199 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836489 2022-05-07T15:53:32.200 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:53:32.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2022-05-07T15:53:32.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:53:33.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:53:33.542 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:53:33.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836489 2022-05-07T15:53:33.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:33.882 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2022-05-07T15:53:33.882 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:33.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:53:33.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2022-05-07T15:53:33.885 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:33.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2022-05-07T15:53:33.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2022-05-07T15:53:33.887 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672968 2022-05-07T15:53:33.887 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:53:34.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2022-05-07T15:53:34.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:34.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509447 2022-05-07T15:53:34.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:34.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:53:34.214 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2022-05-07T15:53:34.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:34.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509447 2022-05-07T15:53:34.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2022-05-07T15:53:34.217 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509447 2022-05-07T15:53:34.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:53:34.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509447 2022-05-07T15:53:34.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:53:34.552 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:53:34.552 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:53:34.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:53:34.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:53:34.956 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:53:34.956 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:53:34.956 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:53:34.957 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:53:34.957 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:53:34.957 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:53:35.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-05-07T15:53:35.286 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:53:35.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:53:35.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:53:35.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-05-07T15:53:35.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-05-07T15:53:35.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-05-07T15:53:35.705 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2022-05-07T15:53:35.705 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2022-05-07T15:53:36.002 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:53:36.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2022-05-07T15:53:36.011 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2022-05-07T15:53:36.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2022-05-07T15:53:36.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:53:36.312 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:53:36.313 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-05-07T15:53:26.109628+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:53:36.313 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:53:36.313 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:53:36.316 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:53:36.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2022-05-07T15:53:36.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2022-05-07T15:53:36.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2022-05-07T15:53:36.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2022-05-07T15:53:36.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2022-05-07T15:53:36.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2022-05-07T15:53:36.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2022-05-07T15:53:36.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2022-05-07T15:53:36.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2022-05-07T15:53:36.329 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2022-05-07T15:53:36.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2022-05-07T15:53:36.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2022-05-07T15:53:36.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2022-05-07T15:53:36.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:36.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2022-05-07T15:53:36.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:36.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2022-05-07T15:53:36.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:36.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2022-05-07T15:53:36.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:36.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2022-05-07T15:53:36.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:36.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2022-05-07T15:53:37.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2022-05-07T15:53:37.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2022-05-07T15:53:37.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2022-05-07T15:53:37.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2022-05-07T15:53:37.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2022-05-07T15:53:37.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2022-05-07T15:53:37.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:37.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2022-05-07T15:53:38.009 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2022-05-07T15:53:38.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2022-05-07T15:53:38.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2022-05-07T15:53:38.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2022-05-07T15:53:38.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2022-05-07T15:53:38.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2022-05-07T15:53:38.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:38.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2022-05-07T15:53:39.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:39.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2022-05-07T15:53:39.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:39.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2022-05-07T15:53:39.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:39.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2022-05-07T15:53:39.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:39.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2022-05-07T15:53:39.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:39.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2022-05-07T15:53:39.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:39.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2022-05-07T15:53:40.009 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2022-05-07T15:53:40.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2022-05-07T15:53:40.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2022-05-07T15:53:40.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2022-05-07T15:53:40.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2022-05-07T15:53:40.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2022-05-07T15:53:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2022-05-07T15:53:40.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:40.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2022-05-07T15:53:41.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:41.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2022-05-07T15:53:41.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:41.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2022-05-07T15:53:41.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:41.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2022-05-07T15:53:41.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:41.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2022-05-07T15:53:41.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:41.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2022-05-07T15:53:41.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:41.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2022-05-07T15:53:42.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.003 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2022-05-07T15:53:42.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2022-05-07T15:53:42.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2022-05-07T15:53:42.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.389 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2022-05-07T15:53:42.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2022-05-07T15:53:42.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2022-05-07T15:53:42.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2022-05-07T15:53:42.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:42.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2022-05-07T15:53:43.066 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2022-05-07T15:53:43.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2022-05-07T15:53:43.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2022-05-07T15:53:43.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2022-05-07T15:53:43.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2022-05-07T15:53:43.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2022-05-07T15:53:43.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:43.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2022-05-07T15:53:44.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2022-05-07T15:53:44.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2022-05-07T15:53:44.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2022-05-07T15:53:44.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2022-05-07T15:53:44.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2022-05-07T15:53:44.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2022-05-07T15:53:44.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2022-05-07T15:53:44.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:44.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2022-05-07T15:53:45.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:45.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2022-05-07T15:53:45.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:45.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2022-05-07T15:53:45.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:45.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2022-05-07T15:53:45.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:45.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2022-05-07T15:53:45.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:45.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2022-05-07T15:53:45.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:45.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2022-05-07T15:53:46.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2022-05-07T15:53:46.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2022-05-07T15:53:46.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2022-05-07T15:53:46.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2022-05-07T15:53:46.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2022-05-07T15:53:46.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2022-05-07T15:53:46.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:46.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2022-05-07T15:53:47.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.044 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2022-05-07T15:53:47.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2022-05-07T15:53:47.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2022-05-07T15:53:47.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2022-05-07T15:53:47.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2022-05-07T15:53:47.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2022-05-07T15:53:47.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:47.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2022-05-07T15:53:48.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2022-05-07T15:53:48.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2022-05-07T15:53:48.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2022-05-07T15:53:48.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2022-05-07T15:53:48.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2022-05-07T15:53:48.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2022-05-07T15:53:48.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:48.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2022-05-07T15:53:49.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2022-05-07T15:53:49.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2022-05-07T15:53:49.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2022-05-07T15:53:49.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2022-05-07T15:53:49.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2022-05-07T15:53:49.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2022-05-07T15:53:49.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2022-05-07T15:53:49.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:49.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2022-05-07T15:53:50.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:50.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2022-05-07T15:53:50.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:50.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2022-05-07T15:53:50.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-05-07T15:53:50.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2022-05-07T15:53:50.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2022-05-07T15:53:50.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2022-05-07T15:53:50.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:53:50.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:53:50.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:53:50.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:53:50.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-05-07T15:53:50.521 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:53:50.522 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:53:50.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:53:50.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:53:50.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:53:50.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:53:50.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:53:50.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:53:50.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:53:50.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:53:51.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:53:51.019 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:53:51.020 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:53:51.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:53:51.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:51.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:53:51.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2022-05-07T15:53:51.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2022-05-07T15:53:51.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2022-05-07T15:53:51.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:51.179 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:53:51.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2022-05-07T15:53:51.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2022-05-07T15:53:51.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2022-05-07T15:53:51.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:53:51.332 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:53:51.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509453 2022-05-07T15:53:51.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509453 2022-05-07T15:53:51.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509453' 2022-05-07T15:53:51.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:51.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2022-05-07T15:53:51.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:51.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:53:51.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2022-05-07T15:53:51.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:51.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2022-05-07T15:53:51.495 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2022-05-07T15:53:51.495 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836495 2022-05-07T15:53:51.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:53:51.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836495 2022-05-07T15:53:51.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:53:52.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:53:52.834 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:53:53.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2022-05-07T15:53:53.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:53.171 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2022-05-07T15:53:53.171 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:53.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:53:53.174 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2022-05-07T15:53:53.176 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:53.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2022-05-07T15:53:53.177 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672974 2022-05-07T15:53:53.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2022-05-07T15:53:53.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:53:53.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2022-05-07T15:53:53.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:53:53.506 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2022-05-07T15:53:53.506 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:53:53.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:53:53.508 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2022-05-07T15:53:53.509 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:53:53.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509453 2022-05-07T15:53:53.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2022-05-07T15:53:53.511 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509453 2022-05-07T15:53:53.512 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:53:53.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509454 -lt 64424509453 2022-05-07T15:53:53.869 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:53:53.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:53:53.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:53:54.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:53:54.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:53:54.289 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:53:54.289 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:53:54.289 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:53:54.289 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:53:54.290 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:53:54.290 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:53:54.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-05-07T15:53:54.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:53:54.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:53:54.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:53:55.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-05-07T15:53:55.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-05-07T15:53:55.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-05-07T15:53:55.024 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2022-05-07T15:53:55.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2022-05-07T15:53:55.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2022-05-07T15:53:55.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2022-05-07T15:53:55.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2022-05-07T15:53:55.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2022-05-07T15:53:55.345 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2022-05-07T15:53:55.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2022-05-07T15:53:55.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:53:55.654 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:53:55.654 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2022-05-07T15:53:26.109628+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:53:55.655 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:53:55.655 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:53:55.656 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:53:55.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:53:55.667 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2022-05-07T15:53:55.667 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-05-07T15:53:55.668 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-05-07T15:53:55.671 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:53:55.671 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:55.671 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:55.672 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.0.asok 2022-05-07T15:53:55.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2022-05-07T15:53:55.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.0.asok config set objectstore_blackhole 1 2022-05-07T15:53:55.747 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:53:55.747 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-05-07T15:53:55.747 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:53:55.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:53:55.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2022-05-07T15:53:55.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-05-07T15:53:55.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-05-07T15:53:55.755 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:53:55.756 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:53:55.756 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:53:55.756 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.2.asok 2022-05-07T15:53:55.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2022-05-07T15:53:55.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.2.asok config set objectstore_blackhole 1 2022-05-07T15:53:55.836 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:53:55.836 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-05-07T15:53:55.836 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:53:55.843 INFO:tasks.workunit.client.0.smithi059.stdout:create new divergent objects 2022-05-07T15:53:55.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2022-05-07T15:53:55.844 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2022-05-07T15:53:55.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-05-07T15:53:55.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-05-07T15:53:55.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2022-05-07T15:53:55.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2022-05-07T15:53:55.847 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent objects 2022-05-07T15:53:55.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2022-05-07T15:53:55.848 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2022-05-07T15:53:55.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2022-05-07T15:53:55.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2022-05-07T15:53:55.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2022-05-07T15:53:55.849 INFO:tasks.workunit.client.0.smithi059.stdout:remove divergent objects 2022-05-07T15:53:55.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2022-05-07T15:53:55.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2022-05-07T15:53:55.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2022-05-07T15:53:55.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-05-07T15:53:55.852 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2022-05-07T15:53:55.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2022-05-07T15:53:55.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-05-07T15:53:55.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2022-05-07T15:53:55.854 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2022-05-07T15:53:55.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2022-05-07T15:53:55.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2022-05-07T15:53:55.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2022-05-07T15:54:05.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2022-05-07T15:54:05.862 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2022-05-07T15:54:05.863 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2022-05-07T15:54:05.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2022-05-07T15:54:05.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:54:06.190 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:54:06.191 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 60778 0 0 16 16 active+clean 2022-05-07T15:53:26.109628+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:54:06.191 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:54:06.191 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:54:06.193 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:54:06.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2022-05-07T15:54:06.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:54:06.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:54:06.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:54:06.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:54:06.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:54:06.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2022-05-07T15:54:06.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2022-05-07T15:54:06.814 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already down. 2022-05-07T15:54:06.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2022-05-07T15:54:06.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2022-05-07T15:54:07.329 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already down. 2022-05-07T15:54:07.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2022-05-07T15:54:07.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2022-05-07T15:54:07.921 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already down. 2022-05-07T15:54:07.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:54:07.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2022-05-07T15:54:08.404 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2022-05-07T15:54:08.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:54:08.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2022-05-07T15:54:08.917 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already out. 2022-05-07T15:54:08.926 INFO:tasks.workunit.client.0.smithi059.stdout:bringing up non_divergent 0 2 2022-05-07T15:54:08.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2022-05-07T15:54:08.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:54:09.263 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:54:09.263 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 60778 0 0 16 16 stale+active+clean 2022-05-07T15:53:26.109628+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:54:09.263 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:54:09.264 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:54:09.266 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:54:09.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:54:09.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2022-05-07T15:54:09.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:54:09.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:54:09.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2022-05-07T15:54:09.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:54:09.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2022-05-07T15:54:09.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:54:09.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:54:09.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:54:09.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:54:09.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-05-07T15:54:09.280 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-05-07T15:54:09.280 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:54:09.280 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:54:09.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:54:09.281 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:54:09.281 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:54:09.282 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:54:09.282 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:54:09.282 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:54:09.283 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:54:09.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:54:09.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:54:09.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:54:09.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:54:09.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:54:09.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:54:09.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:54:09.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:54:09.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:54:09.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:54:09.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2022-05-07T15:54:09.288 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2022-05-07T15:54:09.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2022-05-07T15:54:09.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:54:09.291 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2022-05-07T15:54:09.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2022-05-07T15:54:09.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:54:09.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:54:09.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:54:09.315 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:09.315+0000 7fb2cd0aed80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:54:09.318 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:09.319+0000 7fb2cd0aed80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:54:09.319 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:09.319+0000 7fb2cd0aed80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:54:09.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2022-05-07T15:54:09.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:54:09.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:54:09.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:54:09.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:54:09.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:54:09.607 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:54:09.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:54:09.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:54:09.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:54:09.850 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:09.851+0000 7fb2cd0aed80 -1 Falling back to public interface 2022-05-07T15:54:09.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:54:10.766 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:10.771+0000 7fb2cd0aed80 -1 osd.0 29 log_to_monitors {default=true} 2022-05-07T15:54:10.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:54:10.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:54:10.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:54:10.964 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:54:10.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:54:10.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:54:11.299 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/26695,v1:127.0.0.1:6803/26695] [v2:127.0.0.1:6804/26695,v1:127.0.0.1:6805/26695] exists,up 702a02ae-db92-4105-8037-f4f69248fb71 2022-05-07T15:54:11.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:54:11.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:54:11.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:54:11.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:54:11.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2022-05-07T15:54:11.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:54:11.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:54:11.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2022-05-07T15:54:11.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:54:11.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2022-05-07T15:54:11.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:54:11.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:54:11.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:54:11.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:54:11.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-05-07T15:54:11.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-05-07T15:54:11.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:54:11.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:54:11.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:54:11.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:54:11.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:54:11.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:54:11.308 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:54:11.309 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:54:11.309 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:54:11.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:54:11.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:54:11.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:54:11.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:54:11.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:54:11.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:54:11.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:54:11.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:54:11.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:54:11.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:54:11.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2022-05-07T15:54:11.312 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2022-05-07T15:54:11.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2022-05-07T15:54:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:54:11.314 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2022-05-07T15:54:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2022-05-07T15:54:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:54:11.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:54:11.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:54:11.335 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:11.335+0000 7f9b2ecb1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:54:11.386 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:11.387+0000 7f9b2ecb1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:54:11.387 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:11.387+0000 7f9b2ecb1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:54:11.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2022-05-07T15:54:11.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:54:11.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:54:11.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:54:11.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:54:11.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:54:11.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:54:11.639 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:54:11.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:54:11.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:54:11.926 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:11.927+0000 7f9b2ecb1d80 -1 Falling back to public interface 2022-05-07T15:54:11.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:54:12.914 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:54:12.915+0000 7f9b2ecb1d80 -1 osd.2 29 log_to_monitors {default=true} 2022-05-07T15:54:12.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:54:12.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:54:12.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:54:12.968 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:54:12.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:54:12.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:54:13.409 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/26891,v1:127.0.0.1:6811/26891] [v2:127.0.0.1:6812/26891,v1:127.0.0.1:6813/26891] exists,up b47503be-d6f2-4cab-a385-68acebbdfb7e 2022-05-07T15:54:13.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:54:13.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:54:13.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:54:13.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:54:13.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2022-05-07T15:54:13.985 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already in. 2022-05-07T15:54:13.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:54:13.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2022-05-07T15:54:14.503 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already in. 2022-05-07T15:54:14.512 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2022-05-07T15:54:14.512 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2022-05-07T15:54:14.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:54:14.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:54:14.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:54:14.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:54:14.514 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-05-07T15:54:14.515 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:54:14.515 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:54:14.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:54:14.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:54:14.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:54:14.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:54:14.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:54:14.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:54:14.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:54:14.642 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:54:15.070 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:54:15.071 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:54:15.071 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:54:15.071 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:54:15.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:54:15.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:54:15.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2022-05-07T15:54:15.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2022-05-07T15:54:15.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822659' 2022-05-07T15:54:15.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:54:15.216 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:54:15.323 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-05-07T15:54:15.325 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-05-07T15:54:15.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-05-07T15:54:15.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-05-07T15:54:15.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:54:15.327 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:54:15.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757251 2022-05-07T15:54:15.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757251 2022-05-07T15:54:15.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822659 2-163208757251' 2022-05-07T15:54:15.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:54:15.477 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-154618822659 2022-05-07T15:54:15.477 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:54:15.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:54:15.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822659 2022-05-07T15:54:15.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:54:15.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2022-05-07T15:54:15.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2022-05-07T15:54:15.481 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 154618822659 2022-05-07T15:54:15.482 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:54:15.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 154618822659 2022-05-07T15:54:15.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:54:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:54:16.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:54:17.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822660 -lt 154618822659 2022-05-07T15:54:17.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:54:17.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-163208757251 2022-05-07T15:54:17.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:54:17.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:54:17.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-163208757251 2022-05-07T15:54:17.216 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:54:17.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757251 2022-05-07T15:54:17.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2022-05-07T15:54:17.219 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 163208757251 2022-05-07T15:54:17.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:54:17.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757251 -lt 163208757251 2022-05-07T15:54:17.584 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:54:17.585 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:17.585 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:17.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:54:17.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:18.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:18.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:18.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:18.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:18.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:18.001 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:18.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:18.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:18.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:18.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:18.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:18.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2022-05-07T15:54:18.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2022-05-07T15:54:18.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2022-05-07T15:54:18.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:18.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2022-05-07T15:54:18.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:18.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:18.855 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:18.855 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:18.855 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:18.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:18.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:18.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:19.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:19.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:19.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:19.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:19.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:19.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:19.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:19.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:19.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:19.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:19.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:19.603 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:19.603 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:20.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:20.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:20.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2022-05-07T15:54:20.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:20.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2022-05-07T15:54:20.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:20.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:20.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:20.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:20.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:20.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:20.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:20.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:20.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:20.512 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:20.513 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:20.513 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:20.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:20.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:20.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:20.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:20.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:20.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:20.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:20.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:20.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:21.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:21.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:21.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2022-05-07T15:54:21.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:21.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2022-05-07T15:54:21.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:21.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:21.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:21.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:21.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:21.726 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:21.726 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:21.727 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:22.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:22.044 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:22.045 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:22.045 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:22.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:22.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:22.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:22.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:22.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:22.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:22.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:22.444 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:22.445 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:22.880 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:22.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:22.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2022-05-07T15:54:22.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:22.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2022-05-07T15:54:23.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:23.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:23.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:23.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:23.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:23.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:23.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:23.685 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:24.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:24.019 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:24.019 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:24.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:24.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:24.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:24.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:24.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:24.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:24.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:24.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:24.427 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:24.428 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:24.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:24.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:24.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2022-05-07T15:54:24.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:24.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2022-05-07T15:54:26.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:26.461 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:26.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:26.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:26.463 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:26.463 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:26.463 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:26.463 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:26.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:26.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:26.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:26.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:27.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:27.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:27.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:27.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:27.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:27.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:27.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:27.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:27.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:27.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:27.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:27.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2022-05-07T15:54:27.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:27.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2022-05-07T15:54:30.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:30.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:30.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:30.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:30.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:30.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:30.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:30.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:31.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:31.142 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:31.143 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:31.143 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:31.552 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:31.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:31.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:31.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:31.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:31.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:31.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:31.555 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:31.555 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:31.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:31.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:31.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2022-05-07T15:54:31.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:31.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2022-05-07T15:54:38.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:38.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:38.361 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:38.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:38.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:38.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:38.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:38.363 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:38.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:38.696 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:38.696 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:38.697 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:39.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:39.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:39.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:39.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:39.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:39.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:39.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:39.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:39.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:39.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:39.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:39.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2022-05-07T15:54:39.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:39.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2022-05-07T15:54:46.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:46.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:46.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:46.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:46.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:46.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:46.818 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:46.818 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:47.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:47.145 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:47.145 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:47.145 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:47.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:47.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:47.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:47.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:47.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:47.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:47.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:47.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:47.552 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:47.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:47.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:47.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2022-05-07T15:54:47.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2022-05-07T15:54:48.375 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:54:48.376 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "302839e8-f3a3-4ed8-8439-a1bf19c0a9d7", 2022-05-07T15:54:48.376 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.2.16-103-g63a8334b", 2022-05-07T15:54:48.376 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "63a8334b02d7dac9af12d1cf5f3ef0666b46826e", 2022-05-07T15:54:48.376 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2022-05-07T15:54:48.379858+0000", 2022-05-07T15:54:48.377 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2022-05-07T15:54:48.377 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2022-05-07T15:54:48.377 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2022-05-07T15:54:48.377 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2022-05-07T15:54:48.378 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2022-05-07T15:54:48.378 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:54:48.378 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:54:48.378 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2022-05-07T15:54:48.378 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2022-05-07T15:54:48.379 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.379 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2022-05-07T15:54:48.379 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:54:48.379 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.380 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2022-05-07T15:54:48.380 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:54:48.380 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:54:48.380 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2022-05-07T15:54:48.380 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:54:48.381 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.381 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:54:48.381 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.381 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2022-05-07T15:54:48.382 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.382 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.382 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:54:48.382 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.382 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2022-05-07T15:54:48.383 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:54:48.383 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:54:48.383 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2022-05-07T15:54:48.383 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:54:48.384 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.384 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:54:48.384 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.384 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2022-05-07T15:54:48.384 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.385 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.385 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:54:48.385 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.385 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2022-05-07T15:54:48.386 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:54:48.386 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:54:48.386 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2022-05-07T15:54:48.386 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:54:48.386 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.387 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:54:48.387 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.387 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2022-05-07T15:54:48.387 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.388 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.388 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2022-05-07T15:54:48.388 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.388 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.388 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:54:48.389 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.389 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.389 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2022-05-07T15:54:48.389 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.390 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2022-05-07T15:54:48.390 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2022-05-07T15:54:48.390 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2022-05-07T15:54:48.390 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2022-05-07T15:54:48.390 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "c67bfd60-15c3-4eb2-a02f-223bbda3f2bb", 2022-05-07T15:54:48.391 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:53:00.012314Z", 2022-05-07T15:54:48.391 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2022-05-07T15:53:00.012314Z", 2022-05-07T15:54:48.391 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2022-05-07T15:54:48.391 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2022-05-07T15:54:48.391 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2022-05-07T15:54:48.392 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2022-05-07T15:54:48.392 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2022-05-07T15:54:48.392 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2022-05-07T15:54:48.392 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2022-05-07T15:54:48.393 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2022-05-07T15:54:48.393 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2022-05-07T15:54:48.393 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2022-05-07T15:54:48.393 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.393 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2022-05-07T15:54:48.394 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.395 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2022-05-07T15:54:48.395 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.395 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2022-05-07T15:54:48.396 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2022-05-07T15:54:48.397 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:54:48.397 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.397 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.397 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.398 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2022-05-07T15:54:48.398 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2022-05-07T15:54:48.398 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.403 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.404 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.404 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2022-05-07T15:54:48.404 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2022-05-07T15:54:48.404 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2022-05-07T15:54:48.404 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2022-05-07T15:54:48.405 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.405 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.405 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.405 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2022-05-07T15:54:48.406 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2022-05-07T15:54:48.406 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.406 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2022-05-07T15:54:48.406 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 41, 2022-05-07T15:54:48.406 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "c67bfd60-15c3-4eb2-a02f-223bbda3f2bb", 2022-05-07T15:54:48.407 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2022-05-07T15:53:00.687011+0000", 2022-05-07T15:54:48.407 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:54:14.618748+0000", 2022-05-07T15:54:48.407 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2022-05-07T15:54:13.210821+0000", 2022-05-07T15:54:48.407 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2022-05-07T15:54:14.334263+0000", 2022-05-07T15:54:48.408 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-05-07T15:54:48.408 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2022-05-07T15:54:48.408 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2022-05-07T15:54:48.408 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2022-05-07T15:54:48.408 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2022-05-07T15:54:48.409 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2022-05-07T15:54:48.409 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2022-05-07T15:54:48.409 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2022-05-07T15:54:48.409 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2022-05-07T15:54:48.409 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2022-05-07T15:54:48.410 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.410 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 9, 2022-05-07T15:54:48.410 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2022-05-07T15:54:48.410 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2022-05-07T15:54:48.411 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2022-05-07T15:54:48.411 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2022-05-07T15:54:48.411 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2022-05-07T15:54:48.411 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2022-05-07T15:54:48.411 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2022-05-07T15:54:48.412 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2022-05-07T15:54:48.412 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2022-05-07T15:54:48.412 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2022-05-07T15:54:48.412 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.412 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2022-05-07T15:54:48.413 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2022-05-07T15:54:48.413 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2022-05-07T15:53:25.302949+0000", 2022-05-07T15:54:48.413 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2022-05-07T15:54:48.413 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2022-05-07T15:54:48.413 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2022-05-07T15:54:48.414 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2022-05-07T15:54:48.414 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2022-05-07T15:54:48.414 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2022-05-07T15:54:48.414 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2022-05-07T15:54:48.415 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2022-05-07T15:54:48.415 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2022-05-07T15:54:48.415 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2022-05-07T15:54:48.415 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2022-05-07T15:54:48.415 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2022-05-07T15:54:48.416 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2022-05-07T15:54:48.416 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2022-05-07T15:54:48.416 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2022-05-07T15:54:48.416 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2022-05-07T15:54:48.417 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2022-05-07T15:54:48.417 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2022-05-07T15:54:48.417 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2022-05-07T15:54:48.417 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2022-05-07T15:54:48.417 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.418 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "29", 2022-05-07T15:54:48.418 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2022-05-07T15:54:48.418 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2022-05-07T15:54:48.419 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2022-05-07T15:54:48.419 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2022-05-07T15:54:48.419 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2022-05-07T15:54:48.419 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2022-05-07T15:54:48.419 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2022-05-07T15:54:48.420 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2022-05-07T15:54:48.420 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2022-05-07T15:54:48.420 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2022-05-07T15:54:48.420 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2022-05-07T15:54:48.420 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2022-05-07T15:54:48.421 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2022-05-07T15:54:48.421 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2022-05-07T15:54:48.421 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2022-05-07T15:54:48.421 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2022-05-07T15:54:48.422 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2022-05-07T15:54:48.422 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2022-05-07T15:54:48.422 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-05-07T15:54:48.422 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-05-07T15:54:48.422 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2022-05-07T15:54:48.423 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2022-05-07T15:54:48.423 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2022-05-07T15:54:48.423 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2022-05-07T15:54:48.423 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2022-05-07T15:54:48.424 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2022-05-07T15:54:48.424 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.424 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2022-05-07T15:54:48.424 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2022-05-07T15:54:48.424 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2022-05-07T15:54:48.425 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2022-05-07T15:54:48.425 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2022-05-07T15:54:48.425 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2022-05-07T15:54:48.425 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2022-05-07T15:54:48.425 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2022-05-07T15:54:48.426 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2022-05-07T15:54:48.426 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2022-05-07T15:54:48.426 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2022-05-07T15:54:48.426 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2022-05-07T15:54:48.426 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2022-05-07T15:54:48.427 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.427 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.427 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2022-05-07T15:54:48.427 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.428 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:54:48.428 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "702a02ae-db92-4105-8037-f4f69248fb71", 2022-05-07T15:54:48.428 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2022-05-07T15:54:48.428 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:54:48.428 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:54:48.429 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:54:48.429 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2022-05-07T15:54:48.429 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2022-05-07T15:54:48.429 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 36, 2022-05-07T15:54:48.430 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 40, 2022-05-07T15:54:48.430 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2022-05-07T15:54:48.430 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:54:48.430 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:54:48.430 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.431 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.431 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.431 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2022-05-07T15:54:48.431 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.432 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.432 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.432 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.432 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2022-05-07T15:54:48.432 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.433 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.433 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.433 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.433 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:54:48.433 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.434 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.434 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.434 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2022-05-07T15:54:48.434 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.435 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.435 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.435 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.435 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2022-05-07T15:54:48.435 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.436 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.436 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.436 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.436 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:54:48.436 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.437 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.437 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.437 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2022-05-07T15:54:48.437 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.438 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.438 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.438 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.438 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2022-05-07T15:54:48.438 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.439 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.439 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.439 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.439 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:54:48.439 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.440 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.440 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.440 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2022-05-07T15:54:48.440 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.441 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.441 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.441 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.441 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2022-05-07T15:54:48.441 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:54:48.442 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.442 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.442 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.442 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/26695", 2022-05-07T15:54:48.443 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/26695", 2022-05-07T15:54:48.443 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/26695", 2022-05-07T15:54:48.443 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/26695", 2022-05-07T15:54:48.443 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:54:48.443 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2022-05-07T15:54:48.444 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2022-05-07T15:54:48.444 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.444 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.445 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.445 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:54:48.445 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "389e3986-a3c9-4ea7-b566-1125c8d4e23a", 2022-05-07T15:54:48.445 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2022-05-07T15:54:48.445 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:54:48.446 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:54:48.446 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:54:48.446 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2022-05-07T15:54:48.446 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2022-05-07T15:54:48.447 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2022-05-07T15:54:48.447 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 23, 2022-05-07T15:54:48.447 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2022-05-07T15:54:48.447 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:54:48.447 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:54:48.448 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.448 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.448 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.448 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2022-05-07T15:54:48.448 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.449 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.449 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.449 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.449 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2022-05-07T15:54:48.450 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.450 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.450 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.450 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.450 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:54:48.451 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.451 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.451 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.451 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2022-05-07T15:54:48.452 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.452 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.452 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.452 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.452 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2022-05-07T15:54:48.453 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.453 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.453 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.453 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.454 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:54:48.454 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.454 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.455 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.455 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2022-05-07T15:54:48.455 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.455 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.456 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.456 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.456 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2022-05-07T15:54:48.456 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.456 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.457 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.457 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.457 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:54:48.457 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.458 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.458 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.458 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2022-05-07T15:54:48.458 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.458 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.459 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.459 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.459 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2022-05-07T15:54:48.459 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:54:48.460 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.460 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.460 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.460 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/22285", 2022-05-07T15:54:48.460 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/22285", 2022-05-07T15:54:48.461 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22285", 2022-05-07T15:54:48.461 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22285", 2022-05-07T15:54:48.461 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:54:48.461 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2022-05-07T15:54:48.461 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.462 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.462 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.462 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:54:48.462 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "b47503be-d6f2-4cab-a385-68acebbdfb7e", 2022-05-07T15:54:48.463 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2022-05-07T15:54:48.463 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:54:48.463 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:54:48.463 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:54:48.463 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2022-05-07T15:54:48.464 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2022-05-07T15:54:48.464 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 38, 2022-05-07T15:54:48.464 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2022-05-07T15:54:48.464 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2022-05-07T15:54:48.464 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:54:48.465 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:54:48.465 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.465 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.466 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.466 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2022-05-07T15:54:48.466 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.466 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.467 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.467 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.467 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2022-05-07T15:54:48.467 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.467 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.468 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.468 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.468 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:54:48.468 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.469 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.469 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.469 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2022-05-07T15:54:48.469 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.469 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.470 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.470 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.470 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2022-05-07T15:54:48.470 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.471 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.471 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.471 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.471 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:54:48.472 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.472 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.472 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.472 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2022-05-07T15:54:48.472 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.473 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.473 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.473 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.473 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2022-05-07T15:54:48.474 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.474 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.474 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.474 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.474 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:54:48.475 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:54:48.475 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.475 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:54:48.475 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2022-05-07T15:54:48.476 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.476 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.476 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.476 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:54:48.476 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2022-05-07T15:54:48.477 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:54:48.477 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.477 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.477 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.477 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/26891", 2022-05-07T15:54:48.478 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/26891", 2022-05-07T15:54:48.478 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/26891", 2022-05-07T15:54:48.478 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/26891", 2022-05-07T15:54:48.478 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:54:48.479 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2022-05-07T15:54:48.479 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2022-05-07T15:54:48.479 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.479 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.479 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.480 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2022-05-07T15:54:48.480 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.480 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:54:48.480 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:54:06.651281+0000", 2022-05-07T15:54:48.480 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:54:48.481 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:54:48.481 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:54:48.481 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:54:48.481 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:53:07.919186+0000", 2022-05-07T15:54:48.482 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:54:48.482 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.482 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.482 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:54:48.482 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:54:07.162850+0000", 2022-05-07T15:54:48.483 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:54:48.483 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:54:48.483 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:54:48.483 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:54:48.483 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:53:15.530365+0000", 2022-05-07T15:54:48.484 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:54:48.484 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.486 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.486 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:54:48.486 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:54:07.727278+0000", 2022-05-07T15:54:48.486 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:54:48.487 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:54:48.487 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:54:48.487 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:54:48.487 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:53:21.897954+0000", 2022-05-07T15:54:48.488 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:54:48.488 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.488 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.488 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2022-05-07T15:54:48.488 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2022-05-07T15:54:48.489 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2022-05-07T15:54:48.489 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2022-05-07T15:54:48.489 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2022-05-07T15:54:48.489 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2022-05-07T15:54:48.489 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2022-05-07T15:54:48.490 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2022-05-07T15:54:48.490 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2022-05-07T15:54:48.490 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2022-05-07T15:54:48.490 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2022-05-07T15:54:48.491 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.491 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.491 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2022-05-07T15:54:48.491 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2022-05-07T15:54:48.491 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2022-05-07T15:54:48.492 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2022-05-07T15:54:48.492 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2022-05-07T15:54:48.492 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.492 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2022-05-07T15:54:48.492 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.493 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:54:48.493 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:54:48.493 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/26695,v1:127.0.0.1:6805/26695]", 2022-05-07T15:54:48.493 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:54:48.494 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:54:48.494 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:54:48.494 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:54:48.494 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:54:48.495 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:54:48.495 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:54:48.495 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:54:48.495 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:54:48.495 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2022-05-07T15:54:48.496 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:54:48.496 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:54:48.496 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:54:48.496 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:54:48.497 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:54:48.497 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:54:48.497 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:54:48.497 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:54:48.497 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:54:48.499 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:54:48.500 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:54:48.500 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:54:48.501 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:54:48.501 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:54:48.501 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:54:48.501 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/26695,v1:127.0.0.1:6803/26695]", 2022-05-07T15:54:48.502 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:54:48.502 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/26695,v1:127.0.0.1:6809/26695]", 2022-05-07T15:54:48.502 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/26695,v1:127.0.0.1:6807/26695]", 2022-05-07T15:54:48.502 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:54:48.503 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:54:48.503 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:54:48.503 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:54:48.503 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:54:48.503 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:54:48.504 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:54:48.504 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:54:48.504 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:54:48.504 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2022-05-07T15:54:48.504 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:54:48.505 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:54:48.505 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:54:48.505 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.505 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.506 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:54:48.506 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:54:48.506 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/22285,v1:127.0.0.1:6813/22285]", 2022-05-07T15:54:48.506 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:54:48.507 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:54:48.507 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:54:48.507 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:54:48.507 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:54:48.507 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:54:48.508 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:54:48.508 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:54:48.508 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:54:48.508 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2022-05-07T15:54:48.508 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:54:48.509 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:54:48.509 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:54:48.509 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:54:48.509 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:54:48.509 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:54:48.510 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:54:48.510 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:54:48.510 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:54:48.510 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:54:48.511 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:54:48.511 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:54:48.511 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:54:48.511 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:54:48.511 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:54:48.512 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/22285,v1:127.0.0.1:6811/22285]", 2022-05-07T15:54:48.512 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:54:48.512 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22285,v1:127.0.0.1:6817/22285]", 2022-05-07T15:54:48.512 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22285,v1:127.0.0.1:6815/22285]", 2022-05-07T15:54:48.513 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:54:48.513 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:54:48.513 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:54:48.513 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:54:48.513 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:54:48.514 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:54:48.514 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:54:48.514 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:54:48.514 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:54:48.515 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2022-05-07T15:54:48.515 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:54:48.515 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:54:48.515 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:54:48.515 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.516 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.516 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:54:48.516 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:54:48.516 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/26891,v1:127.0.0.1:6813/26891]", 2022-05-07T15:54:48.517 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:54:48.517 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:54:48.517 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:54:48.517 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:54:48.518 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:54:48.518 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:54:48.518 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:54:48.518 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:54:48.518 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:54:48.519 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2022-05-07T15:54:48.519 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:54:48.519 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:54:48.519 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:54:48.520 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:54:48.520 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:54:48.520 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:54:48.520 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:54:48.521 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:54:48.521 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:54:48.521 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:54:48.521 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:54:48.521 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:54:48.522 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:54:48.522 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:54:48.522 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:54:48.522 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/26891,v1:127.0.0.1:6811/26891]", 2022-05-07T15:54:48.523 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:54:48.523 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/26891,v1:127.0.0.1:6817/26891]", 2022-05-07T15:54:48.523 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/26891,v1:127.0.0.1:6815/26891]", 2022-05-07T15:54:48.524 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:54:48.524 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:54:48.524 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:54:48.524 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:54:48.525 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:54:48.525 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:54:48.525 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:54:48.525 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:54:48.525 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:54:48.526 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2022-05-07T15:54:48.526 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:54:48.526 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:54:48.526 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:54:48.526 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.527 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.527 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_clean_epochs": { 2022-05-07T15:54:48.527 INFO:tasks.workunit.client.0.smithi059.stdout: "min_last_epoch_clean": 0, 2022-05-07T15:54:48.527 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": { 2022-05-07T15:54:48.528 INFO:tasks.workunit.client.0.smithi059.stdout: "per_pool": [] 2022-05-07T15:54:48.528 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.528 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_epochs": [ 2022-05-07T15:54:48.528 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.528 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:54:48.529 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 36 2022-05-07T15:54:48.529 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.529 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.529 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:54:48.530 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 10 2022-05-07T15:54:48.530 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.530 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.530 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:54:48.530 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 39 2022-05-07T15:54:48.531 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.531 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.531 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.531 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2022-05-07T15:54:48.532 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 41, 2022-05-07T15:54:48.532 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2022-05-07T15:54:48.532 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2022-05-07T15:54:48.532 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.532 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:54:48.533 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2022-05-07T15:54:48.533 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:54:48.533 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.533 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.533 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:54:48.534 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2022-05-07T15:54:48.534 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:54:48.534 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.534 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.535 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:54:48.535 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2022-05-07T15:54:48.535 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:54:48.535 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.535 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.536 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2022-05-07T15:54:48.536 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.536 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2022-05-07T15:54:48.536 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2022-05-07T15:54:48.536 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.537 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.537 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:54:48.537 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2022-05-07T15:54:48.537 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.538 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.538 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2022-05-07T15:54:48.538 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2022-05-07T15:54:48.538 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.538 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.539 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2022-05-07T15:54:48.539 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2022-05-07T15:54:48.539 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.539 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.539 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2022-05-07T15:54:48.540 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2022-05-07T15:54:48.540 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.540 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.540 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2022-05-07T15:54:48.541 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2022-05-07T15:54:48.541 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.541 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.541 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2022-05-07T15:54:48.541 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2022-05-07T15:54:48.542 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.542 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.542 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2022-05-07T15:54:48.542 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2022-05-07T15:54:48.543 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.543 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.543 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2022-05-07T15:54:48.543 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2022-05-07T15:54:48.543 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.544 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.544 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2022-05-07T15:54:48.544 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2022-05-07T15:54:48.544 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.544 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.545 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2022-05-07T15:54:48.545 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2022-05-07T15:54:48.545 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.545 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.546 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:54:48.546 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2022-05-07T15:54:48.546 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.546 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.546 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2022-05-07T15:54:48.547 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.547 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2022-05-07T15:54:48.547 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2022-05-07T15:54:48.547 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:54:48.547 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2022-05-07T15:54:48.548 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:54:48.548 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:54:48.548 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:54:48.548 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:54:48.548 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.549 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2022-05-07T15:54:48.549 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:54:48.549 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:54:48.549 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.550 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.550 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.550 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.550 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2022-05-07T15:54:48.550 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2022-05-07T15:54:48.551 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:54:48.551 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2022-05-07T15:54:48.551 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:54:48.551 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:54:48.551 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:54:48.552 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:54:48.552 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.552 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2022-05-07T15:54:48.552 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:54:48.553 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:54:48.553 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.553 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.553 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.554 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.554 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2022-05-07T15:54:48.554 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2022-05-07T15:54:48.554 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:54:48.555 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2022-05-07T15:54:48.555 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:54:48.555 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:54:48.555 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:54:48.555 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:54:48.556 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.556 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:54:48.556 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:54:48.556 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:54:48.556 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.557 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.557 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:54:48.557 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:54:48.557 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2022-05-07T15:54:48.558 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.558 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.558 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:54:48.558 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:54:48.559 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2022-05-07T15:54:48.559 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.559 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.559 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.560 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.560 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2022-05-07T15:54:48.560 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2022-05-07T15:54:48.560 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:54:48.560 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2022-05-07T15:54:48.561 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:54:48.561 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:54:48.561 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:54:48.561 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:54:48.562 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.562 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:54:48.562 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:54:48.562 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:54:48.562 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.563 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.563 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:54:48.563 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:54:48.563 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2022-05-07T15:54:48.563 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.564 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.564 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:54:48.564 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:54:48.564 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2022-05-07T15:54:48.565 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.565 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.565 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.565 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.565 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2022-05-07T15:54:48.566 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.566 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2022-05-07T15:54:48.566 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2022-05-07T15:54:48.567 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2022-05-07T15:54:48.567 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2022-05-07T15:54:48.567 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2022-05-07T15:54:48.568 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2022-05-07T15:54:48.568 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2022-05-07T15:54:48.568 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.569 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2022-05-07T15:54:48.569 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2022-05-07T15:54:48.569 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2022-05-07T15:54:48.569 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.570 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.570 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2022-05-07T15:54:48.570 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2022-05-07T15:54:48.570 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2022-05-07T15:54:48.571 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.571 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.571 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2022-05-07T15:54:48.571 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.572 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:54:48.572 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.573 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.573 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2022-05-07T15:54:48.573 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2022-05-07T15:54:48.573 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2022-05-07T15:54:48.574 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2022-05-07T15:54:48.574 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2022-05-07T15:54:48.574 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2022-05-07T15:54:48.574 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2022-05-07T15:54:48.575 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2022-05-07T15:54:48.575 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2022-05-07T15:54:48.575 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2022-05-07T15:54:48.575 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2022-05-07T15:54:48.575 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2022-05-07T15:54:48.576 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2022-05-07T15:54:48.576 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2022-05-07T15:54:48.577 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2022-05-07T15:54:48.577 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2022-05-07T15:54:48.577 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2022-05-07T15:54:48.577 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2022-05-07T15:54:48.578 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2022-05-07T15:54:48.578 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2022-05-07T15:54:48.578 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2022-05-07T15:54:48.579 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.579 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2022-05-07T15:54:48.580 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.580 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2022-05-07T15:54:48.581 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2022-05-07T15:54:48.581 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2022-05-07T15:54:48.582 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2022-05-07T15:54:48.582 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2022-05-07T15:54:48.582 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2022-05-07T15:54:48.582 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2022-05-07T15:54:48.583 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2022-05-07T15:54:48.583 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2022-05-07T15:54:48.584 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2022-05-07T15:54:48.584 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2022-05-07T15:54:48.584 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2022-05-07T15:54:48.584 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2022-05-07T15:54:48.585 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2022-05-07T15:54:48.585 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2022-05-07T15:54:48.585 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2022-05-07T15:54:48.585 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.586 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.586 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2022-05-07T15:54:48.586 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2022-05-07T15:54:48.587 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2022-05-07T15:54:48.587 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.588 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2022-05-07T15:54:48.588 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2022-05-07T15:54:48.588 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.589 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2022-05-07T15:54:48.589 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2022-05-07T15:54:48.590 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2022-05-07T15:54:48.590 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2022-05-07T15:54:48.590 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2022-05-07T15:54:48.590 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2022-05-07T15:54:48.590 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.591 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2022-05-07T15:54:48.591 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2022-05-07T15:54:48.591 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2022-05-07T15:54:48.591 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2022-05-07T15:54:48.592 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2022-05-07T15:54:48.592 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2022-05-07T15:54:48.592 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 61700, 2022-05-07T15:54:48.592 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2022-05-07T15:54:48.592 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2022-05-07T15:54:48.593 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2022-05-07T15:54:48.593 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2022-05-07T15:54:48.593 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2022-05-07T15:54:48.593 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2022-05-07T15:54:48.594 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2022-05-07T15:54:48.594 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2022-05-07T15:54:48.594 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2022-05-07T15:54:48.594 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2022-05-07T15:54:48.594 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2022-05-07T15:54:48.595 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2022-05-07T15:54:48.595 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2022-05-07T15:54:48.595 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2022-05-07T15:54:48.595 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2022-05-07T15:54:48.596 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2022-05-07T15:54:48.596 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2022-05-07T15:54:48.596 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2022-05-07T15:54:48.596 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2022-05-07T15:54:48.596 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2022-05-07T15:54:48.597 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2022-05-07T15:54:48.597 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2022-05-07T15:54:48.597 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2022-05-07T15:54:48.597 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2022-05-07T15:54:48.598 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2022-05-07T15:54:48.598 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2022-05-07T15:54:48.598 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2022-05-07T15:54:48.599 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2022-05-07T15:54:48.599 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2022-05-07T15:54:48.600 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2022-05-07T15:54:48.600 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2022-05-07T15:54:48.600 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2022-05-07T15:54:48.601 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2022-05-07T15:54:48.601 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2022-05-07T15:54:48.601 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2022-05-07T15:54:48.602 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2022-05-07T15:54:48.602 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2022-05-07T15:54:48.603 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2022-05-07T15:54:48.603 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2022-05-07T15:54:48.603 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.604 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2022-05-07T15:54:48.604 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2022-05-07T15:54:48.604 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2022-05-07T15:54:48.605 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2022-05-07T15:54:48.605 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2022-05-07T15:54:48.605 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2022-05-07T15:54:48.605 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:54:48.605 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:54:48.606 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:54:48.606 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2022-05-07T15:54:48.606 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2022-05-07T15:54:48.606 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.607 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2022-05-07T15:54:48.607 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2022-05-07T15:54:48.607 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2022-05-07T15:54:48.607 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2022-05-07T15:54:48.608 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2022-05-07T15:54:48.608 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.608 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2022-05-07T15:54:48.608 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2022-05-07T15:54:48.608 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2022-05-07T15:54:48.609 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2022-05-07T15:54:48.609 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2022-05-07T15:54:48.609 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2022-05-07T15:54:48.610 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2022-05-07T15:54:48.610 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2022-05-07T15:54:48.610 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179648, 2022-05-07T15:54:48.610 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33664, 2022-05-07T15:54:48.611 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2022-05-07T15:54:48.611 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2022-05-07T15:54:48.611 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393152, 2022-05-07T15:54:48.611 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2022-05-07T15:54:48.612 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2022-05-07T15:54:48.612 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866587648, 2022-05-07T15:54:48.612 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2022-05-07T15:54:48.612 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34471936, 2022-05-07T15:54:48.613 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 457899, 2022-05-07T15:54:48.613 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:54:48.613 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:54:48.613 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:54:48.613 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13678, 2022-05-07T15:54:48.614 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211794 2022-05-07T15:54:48.614 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.614 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2022-05-07T15:54:48.615 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2022-05-07T15:54:48.615 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2022-05-07T15:54:48.616 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2022-05-07T15:54:48.616 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2022-05-07T15:54:48.617 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2022-05-07T15:54:48.617 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2022-05-07T15:54:48.617 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.618 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2022-05-07T15:54:48.618 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2022-05-07T15:54:48.618 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2022-05-07T15:54:48.618 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2022-05-07T15:54:48.619 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2022-05-07T15:54:48.619 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.619 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2022-05-07T15:54:48.620 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2022-05-07T15:54:48.620 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.620 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2022-05-07T15:54:48.620 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2022-05-07T15:54:48.620 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2022-05-07T15:54:48.621 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2022-05-07T15:54:48.621 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2022-05-07T15:54:48.621 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2022-05-07T15:54:48.621 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2022-05-07T15:54:48.622 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2022-05-07T15:54:48.622 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2022-05-07T15:54:48.622 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179648, 2022-05-07T15:54:48.623 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33664, 2022-05-07T15:54:48.623 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2022-05-07T15:54:48.623 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2022-05-07T15:54:48.624 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393152, 2022-05-07T15:54:48.624 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2022-05-07T15:54:48.624 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2022-05-07T15:54:48.624 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866587648, 2022-05-07T15:54:48.625 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2022-05-07T15:54:48.625 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34471936, 2022-05-07T15:54:48.625 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 457899, 2022-05-07T15:54:48.625 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:54:48.626 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:54:48.626 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:54:48.626 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13678, 2022-05-07T15:54:48.626 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211794 2022-05-07T15:54:48.627 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.627 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2022-05-07T15:54:48.627 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2022-05-07T15:54:48.627 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2022-05-07T15:54:48.627 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2022-05-07T15:54:48.628 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2022-05-07T15:54:48.628 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2022-05-07T15:54:48.628 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2022-05-07T15:54:48.629 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.629 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2022-05-07T15:54:48.629 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2022-05-07T15:54:48.630 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2022-05-07T15:54:48.630 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2022-05-07T15:54:48.630 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2022-05-07T15:54:48.631 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.631 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2022-05-07T15:54:48.631 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2022-05-07T15:54:48.631 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.631 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.632 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2022-05-07T15:54:48.632 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.632 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2022-05-07T15:54:48.632 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2022-05-07T15:54:48.633 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2022-05-07T15:54:48.633 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 61700, 2022-05-07T15:54:48.633 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2022-05-07T15:54:48.634 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2022-05-07T15:54:48.634 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2022-05-07T15:54:48.634 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2022-05-07T15:54:48.635 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2022-05-07T15:54:48.635 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2022-05-07T15:54:48.635 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2022-05-07T15:54:48.635 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2022-05-07T15:54:48.636 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2022-05-07T15:54:48.636 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2022-05-07T15:54:48.636 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2022-05-07T15:54:48.636 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2022-05-07T15:54:48.637 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2022-05-07T15:54:48.637 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2022-05-07T15:54:48.637 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2022-05-07T15:54:48.637 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2022-05-07T15:54:48.637 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2022-05-07T15:54:48.638 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2022-05-07T15:54:48.638 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2022-05-07T15:54:48.638 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2022-05-07T15:54:48.639 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2022-05-07T15:54:48.639 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2022-05-07T15:54:48.639 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2022-05-07T15:54:48.639 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2022-05-07T15:54:48.639 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2022-05-07T15:54:48.640 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2022-05-07T15:54:48.640 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2022-05-07T15:54:48.640 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2022-05-07T15:54:48.640 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2022-05-07T15:54:48.641 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2022-05-07T15:54:48.641 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2022-05-07T15:54:48.641 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2022-05-07T15:54:48.642 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2022-05-07T15:54:48.642 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2022-05-07T15:54:48.642 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2022-05-07T15:54:48.642 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2022-05-07T15:54:48.643 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2022-05-07T15:54:48.643 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2022-05-07T15:54:48.643 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2022-05-07T15:54:48.643 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.644 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2022-05-07T15:54:48.644 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2022-05-07T15:54:48.644 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2022-05-07T15:54:48.644 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2022-05-07T15:54:48.644 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19529728, 2022-05-07T15:54:48.645 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 184178, 2022-05-07T15:54:48.645 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:54:48.645 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:54:48.645 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:54:48.646 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2022-05-07T15:54:48.646 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2022-05-07T15:54:48.646 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.646 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2022-05-07T15:54:48.646 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2022-05-07T15:54:48.647 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2022-05-07T15:54:48.647 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2022-05-07T15:54:48.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2022-05-07T15:54:48.647 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.648 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.648 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2022-05-07T15:54:48.648 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.649 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:54:48.649 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 154618822666 2022-05-07T15:54:48.649 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.649 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.650 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:54:48.650 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672977 2022-05-07T15:54:48.650 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.650 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.651 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:54:48.651 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 163208757257 2022-05-07T15:54:48.651 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.651 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.652 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2022-05-07T15:54:48.652 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.652 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2022-05-07T15:54:48.652 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2022-05-07T15:54:48.652 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.653 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.653 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2022-05-07T15:54:48.653 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.653 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:54:48.654 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2022-05-07T15:54:48.654 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2022-05-07T15:54:48.654 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:54:48.654 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.655 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.655 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:54:48.655 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2022-05-07T15:54:48.655 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2022-05-07T15:54:48.656 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:54:48.656 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.656 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.657 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:54:48.657 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2022-05-07T15:54:48.657 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2022-05-07T15:54:48.657 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:54:48.657 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.658 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.658 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2022-05-07T15:54:48.658 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:54:48.659 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2022-05-07T15:54:48.659 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2022-05-07T15:54:48.659 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.659 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:54:48.659 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2022-05-07T15:54:48.660 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 4, 2022-05-07T15:54:48.660 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:54:16.637841+0000", 2022-05-07T15:54:48.660 INFO:tasks.workunit.client.0.smithi059.stdout: "services": { 2022-05-07T15:54:48.660 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": { 2022-05-07T15:54:48.661 INFO:tasks.workunit.client.0.smithi059.stdout: "daemons": { 2022-05-07T15:54:48.661 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": "", 2022-05-07T15:54:48.661 INFO:tasks.workunit.client.0.smithi059.stdout: "0": { 2022-05-07T15:54:48.662 INFO:tasks.workunit.client.0.smithi059.stdout: "start_epoch": 0, 2022-05-07T15:54:48.662 INFO:tasks.workunit.client.0.smithi059.stdout: "start_stamp": "0.000000", 2022-05-07T15:54:48.663 INFO:tasks.workunit.client.0.smithi059.stdout: "gid": 0, 2022-05-07T15:54:48.663 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "(unrecognized address family 0)/0", 2022-05-07T15:54:48.664 INFO:tasks.workunit.client.0.smithi059.stdout: "metadata": {}, 2022-05-07T15:54:48.664 INFO:tasks.workunit.client.0.smithi059.stdout: "task_status": {} 2022-05-07T15:54:48.664 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.665 INFO:tasks.workunit.client.0.smithi059.stdout: "2": { 2022-05-07T15:54:48.665 INFO:tasks.workunit.client.0.smithi059.stdout: "start_epoch": 0, 2022-05-07T15:54:48.666 INFO:tasks.workunit.client.0.smithi059.stdout: "start_stamp": "0.000000", 2022-05-07T15:54:48.666 INFO:tasks.workunit.client.0.smithi059.stdout: "gid": 0, 2022-05-07T15:54:48.667 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "(unrecognized address family 0)/0", 2022-05-07T15:54:48.667 INFO:tasks.workunit.client.0.smithi059.stdout: "metadata": {}, 2022-05-07T15:54:48.667 INFO:tasks.workunit.client.0.smithi059.stdout: "task_status": {} 2022-05-07T15:54:48.668 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.668 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.668 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.668 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.668 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:54:48.669 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2022-05-07T15:54:48.669 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2022-05-07T15:54:48.669 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 347, 2022-05-07T15:54:48.669 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2022-05-07T15:54:48.670 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2022-05-07T15:54:48.670 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:54:48.670 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:54:48.671 INFO:tasks.workunit.client.0.smithi059.stderr:report 1474265913 2022-05-07T15:54:48.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2022-05-07T15:54:48.672 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2022-05-07T15:54:48.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2022-05-07T15:54:48.672 INFO:tasks.workunit.client.0.smithi059.stdout:writing non-divergent object existing_4 2022-05-07T15:54:48.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2022-05-07T15:54:48.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:54:48.693 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:54:48.693 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2022-05-07T15:54:14.916857+0000 29'100 41:120 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:54:48.694 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:54:48.694 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:54:48.696 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:54:48.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2022-05-07T15:54:48.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2022-05-07T15:54:48.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2022-05-07T15:54:48.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:54:48.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:54:48.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:54:48.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:54:48.803 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-05-07T15:54:48.803 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:54:48.804 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:54:48.804 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:54:48.804 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:54:48.804 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:54:48.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:54:48.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:54:48.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:54:48.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:54:48.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:54:49.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:54:49.286 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:54:49.286 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:54:49.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:54:49.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:54:49.287 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:54:49.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822668 2022-05-07T15:54:49.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822668 2022-05-07T15:54:49.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822668' 2022-05-07T15:54:49.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:54:49.472 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:54:49.584 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-05-07T15:54:49.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-05-07T15:54:49.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-05-07T15:54:49.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-05-07T15:54:49.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:54:49.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:54:49.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757260 2022-05-07T15:54:49.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757260 2022-05-07T15:54:49.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822668 2-163208757260' 2022-05-07T15:54:49.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:54:49.744 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-154618822668 2022-05-07T15:54:49.745 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:54:49.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:54:49.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822668 2022-05-07T15:54:49.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:54:49.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822668 2022-05-07T15:54:49.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 154618822668' 2022-05-07T15:54:49.750 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 154618822668 2022-05-07T15:54:49.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:54:50.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822666 -lt 154618822668 2022-05-07T15:54:50.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:54:51.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:54:51.118 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:54:51.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822668 -lt 154618822668 2022-05-07T15:54:51.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:54:51.472 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-163208757260 2022-05-07T15:54:51.472 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:54:51.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:54:51.474 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-163208757260 2022-05-07T15:54:51.475 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:54:51.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757260 2022-05-07T15:54:51.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2022-05-07T15:54:51.477 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 163208757260 2022-05-07T15:54:51.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:54:51.852 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757260 -lt 163208757260 2022-05-07T15:54:51.852 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:54:51.853 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:51.853 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:52.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:54:52.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:52.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:52.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:52.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:52.267 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:52.267 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:52.268 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:52.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:52.580 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:52.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:52.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:52.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:52.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2022-05-07T15:54:52.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2022-05-07T15:54:52.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2022-05-07T15:54:52.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:52.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2022-05-07T15:54:53.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:53.070 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:53.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:53.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:53.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:53.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:53.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:53.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:53.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:53.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:53.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:53.386 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:53.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:53.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:53.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:53.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:53.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:53.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:53.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:53.782 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:53.782 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:54.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:54.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:54.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2022-05-07T15:54:54.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:54.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2022-05-07T15:54:54.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:54.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:54.399 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:54.399 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:54.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:54.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:54.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:54.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:54.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:54.721 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:54.721 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:54.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:55.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:55.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:55.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:55.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:55.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:55.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:55.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:55.124 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:55.124 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:55.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2022-05-07T15:54:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:55.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2022-05-07T15:54:55.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:55.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:55.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:55.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:55.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:55.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:55.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:55.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:56.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:56.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:56.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:56.246 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:56.645 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:56.645 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:56.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:56.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:56.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:56.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:56.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:56.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:56.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:57.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:57.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:57.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2022-05-07T15:54:57.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:57.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2022-05-07T15:54:57.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:54:57.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:54:57.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:54:57.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:54:57.878 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:54:57.878 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:54:57.878 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:54:57.879 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:54:58.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:54:58.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:54:58.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:54:58.209 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:54:58.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:54:58.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:54:58.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:54:58.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:54:58.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:54:58.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:54:58.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:54:58.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:54:58.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:54:59.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:54:59.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:54:59.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2022-05-07T15:54:59.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:54:59.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2022-05-07T15:55:00.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:55:00.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:55:00.638 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:55:00.639 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:55:00.639 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:55:00.639 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:55:00.640 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:55:00.640 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:55:00.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:55:00.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:55:00.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:55:00.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:55:01.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:55:01.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:55:01.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:55:01.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:55:01.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:55:01.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:55:01.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:55:01.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:55:01.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:55:01.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:55:01.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:55:01.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2022-05-07T15:55:01.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:55:01.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2022-05-07T15:55:04.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:55:04.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:55:04.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:55:04.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:55:04.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:55:04.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:55:04.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:55:04.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:55:05.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:55:05.303 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:55:05.304 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:55:05.304 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:55:05.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:55:05.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:55:05.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:55:05.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:55:05.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:55:05.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:55:05.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:55:05.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:55:05.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:55:06.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:55:06.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:55:06.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2022-05-07T15:55:06.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:55:06.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2022-05-07T15:55:12.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:55:12.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:55:12.702 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:55:12.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:55:12.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:55:12.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:55:12.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:55:12.704 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:55:12.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:55:12.911 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:55:12.912 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:55:12.912 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:55:13.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:55:13.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:55:13.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:55:13.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:55:13.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:55:13.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:55:13.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:55:13.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:55:13.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:55:13.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:55:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:55:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2022-05-07T15:55:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-05-07T15:55:13.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2022-05-07T15:55:21.044 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-05-07T15:55:21.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:55:21.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:55:21.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:55:21.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:55:21.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:55:21.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:55:21.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:55:21.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-05-07T15:55:21.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:55:21.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:55:21.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:55:21.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-05-07T15:55:21.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-05-07T15:55:21.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-05-07T15:55:21.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-05-07T15:55:21.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-05-07T15:55:21.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-05-07T15:55:21.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-05-07T15:55:21.789 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-05-07T15:55:21.790 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-05-07T15:55:22.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-05-07T15:55:22.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-05-07T15:55:22.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2022-05-07T15:55:22.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2022-05-07T15:55:22.636 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:55:22.636 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "302839e8-f3a3-4ed8-8439-a1bf19c0a9d7", 2022-05-07T15:55:22.636 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.2.16-103-g63a8334b", 2022-05-07T15:55:22.636 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "63a8334b02d7dac9af12d1cf5f3ef0666b46826e", 2022-05-07T15:55:22.637 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2022-05-07T15:55:22.641963+0000", 2022-05-07T15:55:22.637 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2022-05-07T15:55:22.637 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2022-05-07T15:55:22.637 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2022-05-07T15:55:22.638 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2022-05-07T15:55:22.638 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2022-05-07T15:55:22.638 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:55:22.638 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:55:22.639 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2022-05-07T15:55:22.639 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2022-05-07T15:55:22.639 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.639 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2022-05-07T15:55:22.639 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:55:22.640 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.640 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2022-05-07T15:55:22.641 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:55:22.641 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:55:22.641 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2022-05-07T15:55:22.641 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:55:22.641 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.642 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:55:22.642 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.642 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2022-05-07T15:55:22.642 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.643 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.643 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:55:22.643 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.643 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2022-05-07T15:55:22.644 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:55:22.644 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:55:22.645 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2022-05-07T15:55:22.645 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 2 2022-05-07T15:55:22.646 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.646 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:55:22.646 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.646 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2022-05-07T15:55:22.647 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.647 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.647 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:55:22.647 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.647 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2022-05-07T15:55:22.648 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2022-05-07T15:55:22.648 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2022-05-07T15:55:22.648 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2022-05-07T15:55:22.648 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2022-05-07T15:55:22.649 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.649 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2022-05-07T15:55:22.649 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.649 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2022-05-07T15:55:22.650 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.650 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.650 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2022-05-07T15:55:22.650 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.650 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.651 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2022-05-07T15:55:22.651 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.651 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.651 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2022-05-07T15:55:22.652 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.652 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2022-05-07T15:55:22.652 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2022-05-07T15:55:22.652 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2022-05-07T15:55:22.652 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2022-05-07T15:55:22.653 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "c67bfd60-15c3-4eb2-a02f-223bbda3f2bb", 2022-05-07T15:55:22.653 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:53:00.012314Z", 2022-05-07T15:55:22.653 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2022-05-07T15:53:00.012314Z", 2022-05-07T15:55:22.653 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2022-05-07T15:55:22.654 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2022-05-07T15:55:22.654 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2022-05-07T15:55:22.654 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2022-05-07T15:55:22.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2022-05-07T15:55:22.655 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2022-05-07T15:55:22.655 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2022-05-07T15:55:22.655 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2022-05-07T15:55:22.655 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2022-05-07T15:55:22.656 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2022-05-07T15:55:22.656 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.657 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2022-05-07T15:55:22.657 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.657 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2022-05-07T15:55:22.657 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.657 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2022-05-07T15:55:22.658 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2022-05-07T15:55:22.658 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:55:22.658 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.658 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.659 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.659 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2022-05-07T15:55:22.659 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2022-05-07T15:55:22.659 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.659 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.660 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.660 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2022-05-07T15:55:22.660 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2022-05-07T15:55:22.660 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2022-05-07T15:55:22.661 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2022-05-07T15:55:22.661 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.661 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.661 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.661 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2022-05-07T15:55:22.662 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2022-05-07T15:55:22.662 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.662 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2022-05-07T15:55:22.663 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 41, 2022-05-07T15:55:22.663 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "c67bfd60-15c3-4eb2-a02f-223bbda3f2bb", 2022-05-07T15:55:22.663 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2022-05-07T15:53:00.687011+0000", 2022-05-07T15:55:22.663 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:54:14.618748+0000", 2022-05-07T15:55:22.663 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2022-05-07T15:54:13.210821+0000", 2022-05-07T15:55:22.664 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2022-05-07T15:54:14.334263+0000", 2022-05-07T15:55:22.664 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-05-07T15:55:22.664 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2022-05-07T15:55:22.664 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2022-05-07T15:55:22.665 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2022-05-07T15:55:22.665 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2022-05-07T15:55:22.665 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2022-05-07T15:55:22.665 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2022-05-07T15:55:22.665 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2022-05-07T15:55:22.666 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2022-05-07T15:55:22.666 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2022-05-07T15:55:22.666 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.666 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 9, 2022-05-07T15:55:22.667 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2022-05-07T15:55:22.667 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2022-05-07T15:55:22.667 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2022-05-07T15:55:22.668 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2022-05-07T15:55:22.668 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2022-05-07T15:55:22.668 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2022-05-07T15:55:22.668 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2022-05-07T15:55:22.668 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2022-05-07T15:55:22.669 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2022-05-07T15:55:22.669 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2022-05-07T15:55:22.669 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.669 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2022-05-07T15:55:22.670 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2022-05-07T15:55:22.670 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2022-05-07T15:53:25.302949+0000", 2022-05-07T15:55:22.670 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2022-05-07T15:55:22.670 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2022-05-07T15:55:22.675 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2022-05-07T15:55:22.675 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2022-05-07T15:55:22.676 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2022-05-07T15:55:22.676 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2022-05-07T15:55:22.676 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2022-05-07T15:55:22.676 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2022-05-07T15:55:22.677 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2022-05-07T15:55:22.677 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2022-05-07T15:55:22.677 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2022-05-07T15:55:22.677 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2022-05-07T15:55:22.677 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2022-05-07T15:55:22.678 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2022-05-07T15:55:22.678 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2022-05-07T15:55:22.678 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2022-05-07T15:55:22.678 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2022-05-07T15:55:22.679 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2022-05-07T15:55:22.679 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2022-05-07T15:55:22.679 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2022-05-07T15:55:22.680 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.680 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "29", 2022-05-07T15:55:22.680 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2022-05-07T15:55:22.680 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2022-05-07T15:55:22.680 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2022-05-07T15:55:22.681 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2022-05-07T15:55:22.681 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2022-05-07T15:55:22.681 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2022-05-07T15:55:22.681 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2022-05-07T15:55:22.682 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2022-05-07T15:55:22.682 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2022-05-07T15:55:22.682 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2022-05-07T15:55:22.682 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2022-05-07T15:55:22.682 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2022-05-07T15:55:22.683 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2022-05-07T15:55:22.683 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2022-05-07T15:55:22.683 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2022-05-07T15:55:22.683 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2022-05-07T15:55:22.684 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2022-05-07T15:55:22.684 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2022-05-07T15:55:22.684 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-05-07T15:55:22.684 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-05-07T15:55:22.684 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2022-05-07T15:55:22.685 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2022-05-07T15:55:22.685 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2022-05-07T15:55:22.685 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2022-05-07T15:55:22.685 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2022-05-07T15:55:22.685 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2022-05-07T15:55:22.686 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.686 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2022-05-07T15:55:22.686 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2022-05-07T15:55:22.686 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2022-05-07T15:55:22.687 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2022-05-07T15:55:22.687 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2022-05-07T15:55:22.687 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2022-05-07T15:55:22.687 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2022-05-07T15:55:22.687 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2022-05-07T15:55:22.688 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2022-05-07T15:55:22.688 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2022-05-07T15:55:22.688 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2022-05-07T15:55:22.688 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2022-05-07T15:55:22.689 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2022-05-07T15:55:22.689 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.689 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.689 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2022-05-07T15:55:22.689 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.690 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:55:22.690 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "702a02ae-db92-4105-8037-f4f69248fb71", 2022-05-07T15:55:22.690 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2022-05-07T15:55:22.690 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:55:22.691 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:55:22.691 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:55:22.691 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2022-05-07T15:55:22.691 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2022-05-07T15:55:22.691 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 36, 2022-05-07T15:55:22.692 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 40, 2022-05-07T15:55:22.692 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2022-05-07T15:55:22.692 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:55:22.692 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:55:22.692 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.693 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.693 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.693 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2022-05-07T15:55:22.693 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.694 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.694 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.694 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.694 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2022-05-07T15:55:22.694 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.695 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.695 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.695 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.695 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:55:22.696 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.696 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.696 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.696 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2022-05-07T15:55:22.696 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.697 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.697 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.697 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.697 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2022-05-07T15:55:22.698 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.698 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.698 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.698 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.698 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:55:22.699 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.699 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.699 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.699 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2022-05-07T15:55:22.700 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.700 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.700 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.700 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.700 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2022-05-07T15:55:22.701 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.701 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.701 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.701 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.701 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:55:22.702 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.702 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.702 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.702 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2022-05-07T15:55:22.703 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.703 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.703 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.703 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.703 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2022-05-07T15:55:22.704 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26695 2022-05-07T15:55:22.704 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.704 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.704 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.704 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/26695", 2022-05-07T15:55:22.705 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/26695", 2022-05-07T15:55:22.705 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/26695", 2022-05-07T15:55:22.705 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/26695", 2022-05-07T15:55:22.705 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:55:22.706 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2022-05-07T15:55:22.706 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2022-05-07T15:55:22.706 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.706 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.706 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.707 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:55:22.707 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "389e3986-a3c9-4ea7-b566-1125c8d4e23a", 2022-05-07T15:55:22.707 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2022-05-07T15:55:22.707 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:55:22.708 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:55:22.708 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:55:22.708 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2022-05-07T15:55:22.708 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2022-05-07T15:55:22.708 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2022-05-07T15:55:22.709 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 23, 2022-05-07T15:55:22.709 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2022-05-07T15:55:22.709 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:55:22.709 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:55:22.709 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.710 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.710 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.710 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2022-05-07T15:55:22.710 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.711 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.711 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.711 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.711 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2022-05-07T15:55:22.711 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.712 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.712 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.712 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.712 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:55:22.713 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.713 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.713 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.713 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2022-05-07T15:55:22.713 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.714 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.714 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.714 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.714 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2022-05-07T15:55:22.714 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.715 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.715 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.715 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.715 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:55:22.716 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.716 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.716 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.716 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2022-05-07T15:55:22.717 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.717 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.717 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.717 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.717 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2022-05-07T15:55:22.718 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.718 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.718 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.718 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.718 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:55:22.719 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.719 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.719 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.719 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2022-05-07T15:55:22.720 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.720 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.720 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.720 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.720 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2022-05-07T15:55:22.721 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 22285 2022-05-07T15:55:22.721 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.721 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.721 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.721 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/22285", 2022-05-07T15:55:22.722 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/22285", 2022-05-07T15:55:22.722 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22285", 2022-05-07T15:55:22.722 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22285", 2022-05-07T15:55:22.722 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:55:22.723 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2022-05-07T15:55:22.723 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.723 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.723 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.723 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:55:22.724 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "b47503be-d6f2-4cab-a385-68acebbdfb7e", 2022-05-07T15:55:22.724 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2022-05-07T15:55:22.724 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2022-05-07T15:55:22.724 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2022-05-07T15:55:22.725 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2022-05-07T15:55:22.725 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2022-05-07T15:55:22.725 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2022-05-07T15:55:22.725 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 38, 2022-05-07T15:55:22.725 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2022-05-07T15:55:22.726 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2022-05-07T15:55:22.726 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2022-05-07T15:55:22.726 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2022-05-07T15:55:22.726 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.726 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.727 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.727 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2022-05-07T15:55:22.727 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.727 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.728 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.728 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.728 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2022-05-07T15:55:22.728 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.728 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.729 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.729 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.729 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2022-05-07T15:55:22.729 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.730 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.730 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.730 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2022-05-07T15:55:22.730 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.730 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.731 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.731 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.731 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2022-05-07T15:55:22.731 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.732 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.732 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.732 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.732 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2022-05-07T15:55:22.732 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.733 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.733 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.733 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2022-05-07T15:55:22.733 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.734 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.734 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.734 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.734 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2022-05-07T15:55:22.735 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.735 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.735 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.735 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.735 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2022-05-07T15:55:22.736 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2022-05-07T15:55:22.736 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.736 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2022-05-07T15:55:22.736 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2022-05-07T15:55:22.737 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.737 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.737 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.737 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2022-05-07T15:55:22.737 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2022-05-07T15:55:22.738 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26891 2022-05-07T15:55:22.738 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.738 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.738 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.739 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/26891", 2022-05-07T15:55:22.739 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/26891", 2022-05-07T15:55:22.739 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/26891", 2022-05-07T15:55:22.739 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/26891", 2022-05-07T15:55:22.739 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2022-05-07T15:55:22.740 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2022-05-07T15:55:22.740 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2022-05-07T15:55:22.740 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.740 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.741 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.741 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2022-05-07T15:55:22.741 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.741 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:55:22.741 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:54:06.651281+0000", 2022-05-07T15:55:22.742 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:55:22.742 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:55:22.742 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:55:22.742 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:55:22.743 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:53:07.919186+0000", 2022-05-07T15:55:22.743 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:55:22.743 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.743 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.743 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:55:22.744 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:54:07.162850+0000", 2022-05-07T15:55:22.744 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:55:22.744 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:55:22.744 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:55:22.745 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:55:22.745 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:53:15.530365+0000", 2022-05-07T15:55:22.745 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:55:22.745 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.746 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.746 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:55:22.746 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2022-05-07T15:54:07.727278+0000", 2022-05-07T15:55:22.746 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2022-05-07T15:55:22.747 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2022-05-07T15:55:22.747 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292840890367, 2022-05-07T15:55:22.747 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2022-05-07T15:55:22.747 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2022-05-07T15:53:21.897954+0000", 2022-05-07T15:55:22.748 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2022-05-07T15:55:22.748 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.748 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.748 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2022-05-07T15:55:22.748 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2022-05-07T15:55:22.749 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2022-05-07T15:55:22.749 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2022-05-07T15:55:22.749 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2022-05-07T15:55:22.749 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2022-05-07T15:55:22.750 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2022-05-07T15:55:22.750 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2022-05-07T15:55:22.750 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2022-05-07T15:55:22.750 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2022-05-07T15:55:22.751 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2022-05-07T15:55:22.751 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.751 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.751 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2022-05-07T15:55:22.751 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2022-05-07T15:55:22.752 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2022-05-07T15:55:22.752 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2022-05-07T15:55:22.752 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2022-05-07T15:55:22.752 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.752 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2022-05-07T15:55:22.753 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.753 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:55:22.753 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:55:22.753 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/26695,v1:127.0.0.1:6805/26695]", 2022-05-07T15:55:22.754 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:55:22.754 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:55:22.754 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:55:22.754 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:55:22.754 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:55:22.755 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:55:22.755 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:55:22.755 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:55:22.755 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:55:22.755 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2022-05-07T15:55:22.756 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:55:22.756 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:55:22.756 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:55:22.756 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:55:22.757 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:55:22.757 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:55:22.757 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:55:22.757 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:55:22.757 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:55:22.758 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:55:22.758 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:55:22.758 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:55:22.758 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:55:22.759 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:55:22.759 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:55:22.759 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/26695,v1:127.0.0.1:6803/26695]", 2022-05-07T15:55:22.759 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:55:22.760 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/26695,v1:127.0.0.1:6809/26695]", 2022-05-07T15:55:22.760 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/26695,v1:127.0.0.1:6807/26695]", 2022-05-07T15:55:22.760 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:55:22.760 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:55:22.760 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:55:22.761 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:55:22.761 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:55:22.761 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:55:22.761 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:55:22.762 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:55:22.762 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:55:22.762 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2022-05-07T15:55:22.762 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:55:22.762 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:55:22.763 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:55:22.763 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.763 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.763 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:55:22.764 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:55:22.764 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/22285,v1:127.0.0.1:6813/22285]", 2022-05-07T15:55:22.764 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:55:22.764 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:55:22.764 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:55:22.765 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:55:22.765 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:55:22.765 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:55:22.765 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:55:22.765 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:55:22.766 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:55:22.766 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2022-05-07T15:55:22.766 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:55:22.766 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:55:22.767 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:55:22.767 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:55:22.767 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:55:22.767 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:55:22.767 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:55:22.768 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:55:22.768 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:55:22.768 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:55:22.768 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:55:22.769 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:55:22.769 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:55:22.769 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:55:22.769 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:55:22.769 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/22285,v1:127.0.0.1:6811/22285]", 2022-05-07T15:55:22.770 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:55:22.770 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22285,v1:127.0.0.1:6817/22285]", 2022-05-07T15:55:22.770 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22285,v1:127.0.0.1:6815/22285]", 2022-05-07T15:55:22.770 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:55:22.771 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:55:22.771 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:55:22.771 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:55:22.771 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:55:22.771 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:55:22.772 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:55:22.772 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:55:22.772 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:55:22.772 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2022-05-07T15:55:22.772 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:55:22.773 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:55:22.773 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:55:22.773 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.773 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.774 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:55:22.774 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2022-05-07T15:55:22.774 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/26891,v1:127.0.0.1:6813/26891]", 2022-05-07T15:55:22.774 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2022-05-07T15:55:22.774 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2022-05-07T15:55:22.775 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2022-05-07T15:55:22.775 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2022-05-07T15:55:22.775 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2022-05-07T15:55:22.776 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2022-05-07T15:55:22.776 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2022-05-07T15:55:22.776 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2022-05-07T15:55:22.776 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-05-07T15:55:22.777 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2022-05-07T15:55:22.777 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2022-05-07T15:55:22.777 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2022-05-07T15:55:22.777 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2022-05-07T15:55:22.777 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2022-05-07T15:55:22.778 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2022-05-07T15:55:22.778 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.2.16-103-g63a8334b (63a8334b02d7dac9af12d1cf5f3ef0666b46826e) octopus (stable)", 2022-05-07T15:55:22.778 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.2.16-103-g63a8334b", 2022-05-07T15:55:22.778 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2022-05-07T15:55:22.778 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2022-05-07T15:55:22.779 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2022-05-07T15:55:22.779 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-05-07T15:55:22.779 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2022-05-07T15:55:22.779 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "ubuntu", 2022-05-07T15:55:22.780 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2022-05-07T15:55:22.780 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "20.04", 2022-05-07T15:55:22.780 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/26891,v1:127.0.0.1:6811/26891]", 2022-05-07T15:55:22.780 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2022-05-07T15:55:22.781 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/26891,v1:127.0.0.1:6817/26891]", 2022-05-07T15:55:22.781 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/26891,v1:127.0.0.1:6815/26891]", 2022-05-07T15:55:22.781 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2022-05-07T15:55:22.781 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2022-05-07T15:55:22.781 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022", 2022-05-07T15:55:22.782 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "5.4.0-109-generic", 2022-05-07T15:55:22.782 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "2097148", 2022-05-07T15:55:22.782 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32773272", 2022-05-07T15:55:22.782 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-05-07T15:55:22.782 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2022-05-07T15:55:22.783 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2022-05-07T15:55:22.783 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2022-05-07T15:55:22.783 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2022-05-07T15:55:22.783 INFO:tasks.workunit.client.0.smithi059.stdout: "osdspec_affinity": "", 2022-05-07T15:55:22.784 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2022-05-07T15:55:22.784 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.784 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.784 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_clean_epochs": { 2022-05-07T15:55:22.784 INFO:tasks.workunit.client.0.smithi059.stdout: "min_last_epoch_clean": 0, 2022-05-07T15:55:22.785 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": { 2022-05-07T15:55:22.785 INFO:tasks.workunit.client.0.smithi059.stdout: "per_pool": [] 2022-05-07T15:55:22.785 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.785 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_epochs": [ 2022-05-07T15:55:22.786 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.786 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:55:22.786 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 36 2022-05-07T15:55:22.786 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.786 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.787 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:55:22.787 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 10 2022-05-07T15:55:22.787 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.787 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.788 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:55:22.788 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 39 2022-05-07T15:55:22.788 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.788 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.789 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.789 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2022-05-07T15:55:22.789 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 41, 2022-05-07T15:55:22.789 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2022-05-07T15:55:22.790 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2022-05-07T15:55:22.790 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.790 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:55:22.790 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2022-05-07T15:55:22.790 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:55:22.791 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.791 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.791 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:55:22.791 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2022-05-07T15:55:22.792 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:55:22.792 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.792 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.792 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:55:22.793 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2022-05-07T15:55:22.793 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2022-05-07T15:55:22.793 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.793 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.793 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2022-05-07T15:55:22.794 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.794 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2022-05-07T15:55:22.794 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2022-05-07T15:55:22.794 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.795 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.795 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:55:22.795 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2022-05-07T15:55:22.795 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.795 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.796 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2022-05-07T15:55:22.796 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2022-05-07T15:55:22.796 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.796 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.797 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2022-05-07T15:55:22.797 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2022-05-07T15:55:22.797 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.797 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.797 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2022-05-07T15:55:22.798 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2022-05-07T15:55:22.798 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.798 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.798 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2022-05-07T15:55:22.799 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2022-05-07T15:55:22.799 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.799 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.799 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2022-05-07T15:55:22.800 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2022-05-07T15:55:22.800 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.800 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.800 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2022-05-07T15:55:22.800 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2022-05-07T15:55:22.801 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.801 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.801 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2022-05-07T15:55:22.801 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2022-05-07T15:55:22.801 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.802 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.802 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2022-05-07T15:55:22.802 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2022-05-07T15:55:22.802 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.803 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.803 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2022-05-07T15:55:22.803 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2022-05-07T15:55:22.803 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.804 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.804 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:55:22.804 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2022-05-07T15:55:22.804 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.804 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.805 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2022-05-07T15:55:22.805 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.805 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2022-05-07T15:55:22.805 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2022-05-07T15:55:22.806 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:55:22.806 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2022-05-07T15:55:22.806 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:55:22.806 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:55:22.806 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:55:22.807 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:55:22.807 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.807 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2022-05-07T15:55:22.807 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:55:22.808 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:55:22.808 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.808 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.808 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.809 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.809 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2022-05-07T15:55:22.809 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2022-05-07T15:55:22.809 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2022-05-07T15:55:22.809 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2022-05-07T15:55:22.810 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:55:22.810 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:55:22.810 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:55:22.810 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:55:22.810 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.811 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2022-05-07T15:55:22.811 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:55:22.811 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:55:22.811 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.812 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.812 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.812 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.812 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2022-05-07T15:55:22.813 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2022-05-07T15:55:22.813 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:55:22.813 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2022-05-07T15:55:22.813 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:55:22.813 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:55:22.814 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:55:22.814 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:55:22.814 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.814 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:55:22.815 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:55:22.815 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:55:22.815 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.815 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.816 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:55:22.816 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:55:22.816 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2022-05-07T15:55:22.816 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.816 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.817 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:55:22.817 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:55:22.817 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2022-05-07T15:55:22.817 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.818 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.818 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.818 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.818 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2022-05-07T15:55:22.819 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2022-05-07T15:55:22.819 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2022-05-07T15:55:22.819 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2022-05-07T15:55:22.819 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2022-05-07T15:55:22.820 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2022-05-07T15:55:22.820 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2022-05-07T15:55:22.820 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2022-05-07T15:55:22.820 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.820 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2022-05-07T15:55:22.821 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:55:22.821 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2022-05-07T15:55:22.821 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.821 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.821 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2022-05-07T15:55:22.822 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:55:22.822 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2022-05-07T15:55:22.822 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.822 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.823 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2022-05-07T15:55:22.823 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2022-05-07T15:55:22.823 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2022-05-07T15:55:22.823 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.824 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.824 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.824 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.824 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2022-05-07T15:55:22.824 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.825 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2022-05-07T15:55:22.825 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2022-05-07T15:55:22.825 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2022-05-07T15:55:22.825 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2022-05-07T15:55:22.826 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2022-05-07T15:55:22.826 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2022-05-07T15:55:22.826 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2022-05-07T15:55:22.826 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.827 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2022-05-07T15:55:22.827 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2022-05-07T15:55:22.827 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2022-05-07T15:55:22.827 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.827 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.828 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2022-05-07T15:55:22.828 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2022-05-07T15:55:22.828 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2022-05-07T15:55:22.828 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.829 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.829 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2022-05-07T15:55:22.829 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.829 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2022-05-07T15:55:22.829 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.830 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.830 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2022-05-07T15:55:22.830 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2022-05-07T15:55:22.830 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2022-05-07T15:55:22.830 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2022-05-07T15:55:22.831 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2022-05-07T15:55:22.831 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2022-05-07T15:55:22.831 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2022-05-07T15:55:22.831 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2022-05-07T15:55:22.832 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2022-05-07T15:55:22.832 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2022-05-07T15:55:22.832 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2022-05-07T15:55:22.832 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2022-05-07T15:55:22.832 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2022-05-07T15:55:22.833 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2022-05-07T15:55:22.833 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2022-05-07T15:55:22.833 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2022-05-07T15:55:22.833 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2022-05-07T15:55:22.834 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2022-05-07T15:55:22.834 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2022-05-07T15:55:22.834 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2022-05-07T15:55:22.834 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2022-05-07T15:55:22.834 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.835 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2022-05-07T15:55:22.835 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.835 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2022-05-07T15:55:22.835 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2022-05-07T15:55:22.836 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2022-05-07T15:55:22.836 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2022-05-07T15:55:22.836 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2022-05-07T15:55:22.836 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2022-05-07T15:55:22.836 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2022-05-07T15:55:22.837 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2022-05-07T15:55:22.837 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2022-05-07T15:55:22.837 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2022-05-07T15:55:22.837 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2022-05-07T15:55:22.837 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2022-05-07T15:55:22.838 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2022-05-07T15:55:22.838 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2022-05-07T15:55:22.838 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2022-05-07T15:55:22.838 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2022-05-07T15:55:22.838 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.839 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.839 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2022-05-07T15:55:22.839 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2022-05-07T15:55:22.839 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2022-05-07T15:55:22.839 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.840 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2022-05-07T15:55:22.840 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2022-05-07T15:55:22.840 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.840 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2022-05-07T15:55:22.841 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2022-05-07T15:55:22.841 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2022-05-07T15:55:22.841 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2022-05-07T15:55:22.841 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2022-05-07T15:55:22.841 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2022-05-07T15:55:22.842 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.842 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2022-05-07T15:55:22.842 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2022-05-07T15:55:22.842 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2022-05-07T15:55:22.843 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2022-05-07T15:55:22.843 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2022-05-07T15:55:22.843 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2022-05-07T15:55:22.843 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 61856, 2022-05-07T15:55:22.843 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2022-05-07T15:55:22.844 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2022-05-07T15:55:22.844 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2022-05-07T15:55:22.844 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2022-05-07T15:55:22.844 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2022-05-07T15:55:22.844 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2022-05-07T15:55:22.845 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2022-05-07T15:55:22.845 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2022-05-07T15:55:22.845 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2022-05-07T15:55:22.845 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2022-05-07T15:55:22.845 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2022-05-07T15:55:22.846 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2022-05-07T15:55:22.846 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2022-05-07T15:55:22.846 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2022-05-07T15:55:22.846 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2022-05-07T15:55:22.847 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2022-05-07T15:55:22.847 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2022-05-07T15:55:22.847 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2022-05-07T15:55:22.847 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2022-05-07T15:55:22.847 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2022-05-07T15:55:22.848 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2022-05-07T15:55:22.848 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2022-05-07T15:55:22.848 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2022-05-07T15:55:22.848 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2022-05-07T15:55:22.848 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2022-05-07T15:55:22.849 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2022-05-07T15:55:22.849 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2022-05-07T15:55:22.849 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2022-05-07T15:55:22.849 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2022-05-07T15:55:22.849 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2022-05-07T15:55:22.850 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2022-05-07T15:55:22.850 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2022-05-07T15:55:22.850 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2022-05-07T15:55:22.850 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2022-05-07T15:55:22.850 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2022-05-07T15:55:22.851 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2022-05-07T15:55:22.851 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2022-05-07T15:55:22.851 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2022-05-07T15:55:22.851 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2022-05-07T15:55:22.852 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.852 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2022-05-07T15:55:22.852 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2022-05-07T15:55:22.852 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2022-05-07T15:55:22.852 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2022-05-07T15:55:22.853 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2022-05-07T15:55:22.853 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2022-05-07T15:55:22.853 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:55:22.853 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:55:22.853 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:55:22.854 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2022-05-07T15:55:22.854 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2022-05-07T15:55:22.854 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.854 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 17, 2022-05-07T15:55:22.855 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 17, 2022-05-07T15:55:22.855 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2022-05-07T15:55:22.855 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2022-05-07T15:55:22.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2022-05-07T15:55:22.856 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.856 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2022-05-07T15:55:22.856 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2022-05-07T15:55:22.856 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2022-05-07T15:55:22.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2022-05-07T15:55:22.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2022-05-07T15:55:22.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2022-05-07T15:55:22.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2022-05-07T15:55:22.857 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2022-05-07T15:55:22.858 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179648, 2022-05-07T15:55:22.858 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33664, 2022-05-07T15:55:22.858 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2022-05-07T15:55:22.858 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2022-05-07T15:55:22.858 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393152, 2022-05-07T15:55:22.859 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2022-05-07T15:55:22.859 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2022-05-07T15:55:22.859 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866587648, 2022-05-07T15:55:22.859 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2022-05-07T15:55:22.860 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34471936, 2022-05-07T15:55:22.860 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 458211, 2022-05-07T15:55:22.860 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:55:22.860 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:55:22.860 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:55:22.861 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13678, 2022-05-07T15:55:22.861 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211794 2022-05-07T15:55:22.861 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.861 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2022-05-07T15:55:22.862 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2022-05-07T15:55:22.862 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2022-05-07T15:55:22.862 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2022-05-07T15:55:22.862 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2022-05-07T15:55:22.862 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2022-05-07T15:55:22.863 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2022-05-07T15:55:22.863 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.863 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2022-05-07T15:55:22.863 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2022-05-07T15:55:22.864 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2022-05-07T15:55:22.864 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2022-05-07T15:55:22.864 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2022-05-07T15:55:22.864 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.865 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2022-05-07T15:55:22.865 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2022-05-07T15:55:22.865 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.865 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2022-05-07T15:55:22.865 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2022-05-07T15:55:22.866 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2022-05-07T15:55:22.866 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2022-05-07T15:55:22.866 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2022-05-07T15:55:22.866 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2022-05-07T15:55:22.867 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2022-05-07T15:55:22.867 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2022-05-07T15:55:22.867 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2022-05-07T15:55:22.867 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179648, 2022-05-07T15:55:22.867 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33664, 2022-05-07T15:55:22.868 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2022-05-07T15:55:22.868 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2022-05-07T15:55:22.868 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393152, 2022-05-07T15:55:22.868 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2022-05-07T15:55:22.869 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2022-05-07T15:55:22.869 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866587648, 2022-05-07T15:55:22.869 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2022-05-07T15:55:22.869 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34471936, 2022-05-07T15:55:22.869 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 458211, 2022-05-07T15:55:22.870 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:55:22.870 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:55:22.870 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:55:22.870 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13678, 2022-05-07T15:55:22.871 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211794 2022-05-07T15:55:22.871 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.871 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2022-05-07T15:55:22.871 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2022-05-07T15:55:22.871 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2022-05-07T15:55:22.872 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2022-05-07T15:55:22.872 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2022-05-07T15:55:22.872 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2022-05-07T15:55:22.872 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2022-05-07T15:55:22.873 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.873 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2022-05-07T15:55:22.873 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2022-05-07T15:55:22.873 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2022-05-07T15:55:22.873 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2022-05-07T15:55:22.874 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2022-05-07T15:55:22.874 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.874 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2022-05-07T15:55:22.874 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2022-05-07T15:55:22.875 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.875 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.875 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2022-05-07T15:55:22.875 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.876 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2022-05-07T15:55:22.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2022-05-07T15:55:22.876 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2022-05-07T15:55:22.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 61856, 2022-05-07T15:55:22.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2022-05-07T15:55:22.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2022-05-07T15:55:22.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2022-05-07T15:55:22.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2022-05-07T15:55:22.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2022-05-07T15:55:22.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2022-05-07T15:55:22.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2022-05-07T15:55:22.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2022-05-07T15:55:22.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2022-05-07T15:55:22.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2022-05-07T15:55:22.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2022-05-07T15:55:22.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2022-05-07T15:55:22.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2022-05-07T15:55:22.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2022-05-07T15:55:22.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2022-05-07T15:55:22.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2022-05-07T15:55:22.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2022-05-07T15:55:22.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2022-05-07T15:55:22.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2022-05-07T15:55:22.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2022-05-07T15:55:22.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2022-05-07T15:55:22.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2022-05-07T15:55:22.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2022-05-07T15:55:22.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2022-05-07T15:55:22.882 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2022-05-07T15:55:22.882 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2022-05-07T15:55:22.882 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2022-05-07T15:55:22.882 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2022-05-07T15:55:22.882 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2022-05-07T15:55:22.883 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2022-05-07T15:55:22.883 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2022-05-07T15:55:22.883 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2022-05-07T15:55:22.883 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2022-05-07T15:55:22.884 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2022-05-07T15:55:22.884 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2022-05-07T15:55:22.884 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2022-05-07T15:55:22.884 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2022-05-07T15:55:22.884 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2022-05-07T15:55:22.885 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2022-05-07T15:55:22.885 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.885 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2022-05-07T15:55:22.885 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2022-05-07T15:55:22.886 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2022-05-07T15:55:22.886 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2022-05-07T15:55:22.886 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19529728, 2022-05-07T15:55:22.886 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 184490, 2022-05-07T15:55:22.887 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2022-05-07T15:55:22.887 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2022-05-07T15:55:22.887 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2022-05-07T15:55:22.887 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2022-05-07T15:55:22.887 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2022-05-07T15:55:22.888 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.888 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 17, 2022-05-07T15:55:22.888 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 17, 2022-05-07T15:55:22.888 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2022-05-07T15:55:22.888 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2022-05-07T15:55:22.889 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2022-05-07T15:55:22.889 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.889 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.889 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2022-05-07T15:55:22.890 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.890 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:55:22.890 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 154618822674 2022-05-07T15:55:22.890 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.890 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.891 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:55:22.891 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672977 2022-05-07T15:55:22.891 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.891 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.892 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:55:22.892 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 163208757266 2022-05-07T15:55:22.892 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.892 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.892 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2022-05-07T15:55:22.893 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.893 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2022-05-07T15:55:22.893 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2022-05-07T15:55:22.893 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.893 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.894 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2022-05-07T15:55:22.894 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.894 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2022-05-07T15:55:22.894 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2022-05-07T15:55:22.895 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2022-05-07T15:55:22.895 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:55:22.895 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.895 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.895 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2022-05-07T15:55:22.896 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2022-05-07T15:55:22.896 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2022-05-07T15:55:22.896 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:55:22.896 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.897 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.897 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2022-05-07T15:55:22.897 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2022-05-07T15:55:22.897 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2022-05-07T15:55:22.897 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2022-05-07T15:55:22.898 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.898 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.898 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2022-05-07T15:55:22.898 INFO:tasks.workunit.client.0.smithi059.stdout: { 2022-05-07T15:55:22.899 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2022-05-07T15:55:22.899 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2022-05-07T15:55:22.899 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.899 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2022-05-07T15:55:22.900 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2022-05-07T15:55:22.900 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 6, 2022-05-07T15:55:22.900 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2022-05-07T15:55:14.654416+0000", 2022-05-07T15:55:22.900 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2022-05-07T15:55:22.900 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2022-05-07T15:55:22.901 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2022-05-07T15:55:22.901 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2022-05-07T15:55:22.901 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 432, 2022-05-07T15:55:22.901 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2022-05-07T15:55:22.901 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2022-05-07T15:55:22.902 INFO:tasks.workunit.client.0.smithi059.stdout: } 2022-05-07T15:55:22.902 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:55:22.903 INFO:tasks.workunit.client.0.smithi059.stderr:report 1274265063 2022-05-07T15:55:22.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2022-05-07T15:55:22.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2022-05-07T15:55:22.904 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery 2022-05-07T15:55:22.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:22.982 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:22.983 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61856 0 0 17 17 active+undersized+degraded 2022-05-07T15:54:14.916857+0000 41'101 41:121 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:22.983 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:22.983 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:22.985 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:22.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:55:22.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2022-05-07T15:55:22.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-05-07T15:55:22.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-05-07T15:55:22.996 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:55:22.996 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:55:22.996 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:55:22.996 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.0.asok 2022-05-07T15:55:22.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2022-05-07T15:55:22.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.0.asok set_recovery_delay 100000 2022-05-07T15:55:23.075 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-05-07T15:55:23.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2022-05-07T15:55:23.083 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2022-05-07T15:55:23.083 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-05-07T15:55:23.084 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-05-07T15:55:23.085 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:55:23.085 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:55:23.085 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:55:23.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.2.asok 2022-05-07T15:55:23.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2022-05-07T15:55:23.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.2.asok set_recovery_delay 100000 2022-05-07T15:55:23.166 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-05-07T15:55:23.172 INFO:tasks.workunit.client.0.smithi059.stdout:revive divergent 1 2022-05-07T15:55:23.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2022-05-07T15:55:23.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:23.482 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:23.483 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61856 0 0 17 17 active+undersized+degraded 2022-05-07T15:54:14.916857+0000 41'101 41:121 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:23.483 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:23.483 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:23.486 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:23.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2022-05-07T15:55:24.107 INFO:tasks.workunit.client.0.smithi059.stderr:noup is set 2022-05-07T15:55:24.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2022-05-07T15:55:24.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:55:24.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:55:24.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-05-07T15:55:24.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:55:24.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:55:24.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:55:24.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:55:24.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:55:24.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:55:24.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:55:24.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:55:24.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:55:24.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:55:24.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:55:24.124 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:55:24.125 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:55:24.125 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:55:24.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:55:24.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:55:24.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:55:24.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:55:24.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:55:24.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:55:24.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:55:24.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:55:24.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:55:24.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:55:24.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:55:24.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:55:24.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:55:24.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:55:24.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-05-07T15:55:24.131 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:55:24.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:55:24.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-05-07T15:55:24.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-05-07T15:55:24.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:55:24.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:55:24.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:55:24.162 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:24.163+0000 7f29e792dd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:55:24.164 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:24.167+0000 7f29e792dd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:55:24.171 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:24.171+0000 7f29e792dd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:55:24.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2022-05-07T15:55:24.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2022-05-07T15:55:24.710 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:24.711+0000 7f29e792dd80 -1 Falling back to public interface 2022-05-07T15:55:25.690 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:25.691+0000 7f29e792dd80 -1 osd.1 29 log_to_monitors {default=true} 2022-05-07T15:55:29.467 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery divergent 2022-05-07T15:55:29.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2022-05-07T15:55:29.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:29.802 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:29.802 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 61856 0 0 17 17 active+undersized+degraded 2022-05-07T15:54:14.916857+0000 41'101 42:123 [0,2] 0 [0,2] 0 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:29.802 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:29.803 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:29.805 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:29.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2022-05-07T15:55:29.814 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-05-07T15:55:29.814 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-05-07T15:55:29.815 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:55:29.815 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:55:29.816 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:55:29.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.1.asok 2022-05-07T15:55:29.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2022-05-07T15:55:29.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.1.asok set_recovery_delay 100000 2022-05-07T15:55:29.902 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-05-07T15:55:29.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2022-05-07T15:55:30.540 INFO:tasks.workunit.client.0.smithi059.stderr:noup is unset 2022-05-07T15:55:30.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2022-05-07T15:55:30.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:55:30.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:55:30.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:55:30.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:55:30.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:55:30.557 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:55:30.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:55:30.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:55:30.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:55:30.932 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/26695,v1:127.0.0.1:6803/26695] [v2:127.0.0.1:6804/26695,v1:127.0.0.1:6805/26695] exists,up 702a02ae-db92-4105-8037-f4f69248fb71 2022-05-07T15:55:30.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:55:30.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:55:30.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:55:30.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2022-05-07T15:55:30.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:55:30.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:55:30.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:55:30.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:55:30.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:55:30.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:55:30.938 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:55:30.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:55:30.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:55:31.283 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/29376,v1:127.0.0.1:6819/29376] [v2:127.0.0.1:6820/29376,v1:127.0.0.1:6821/29376] exists,up 389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:55:31.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:55:31.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:55:31.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:55:31.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2022-05-07T15:55:31.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:55:31.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:55:31.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:55:31.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:55:31.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:55:31.287 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:55:31.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:55:31.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:55:31.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:55:31.668 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/26891,v1:127.0.0.1:6811/26891] [v2:127.0.0.1:6812/26891,v1:127.0.0.1:6813/26891] exists,up b47503be-d6f2-4cab-a385-68acebbdfb7e 2022-05-07T15:55:31.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:55:31.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:55:31.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:55:31.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:31.975 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:31.976 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 60778 0 0 16 16 peering 2022-05-07T15:55:30.640617+0000 29'106 46:117 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:31.976 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:31.976 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:31.978 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:31.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2022-05-07T15:55:31.991 INFO:tasks.workunit.client.0.smithi059.stdout:wait for peering 2022-05-07T15:55:31.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:32.300 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:32.300 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 60778 0 0 16 16 peering 2022-05-07T15:55:30.640617+0000 29'106 46:117 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:32.300 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:32.301 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:32.302 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:32.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2022-05-07T15:55:32.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2022-05-07T15:55:32.442 INFO:tasks.workunit.client.0.smithi059.stdout:killing divergent 1 2022-05-07T15:55:32.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:32.752 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:32.753 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 60778 0 0 16 16 peering 2022-05-07T15:55:30.640617+0000 29'106 46:117 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:32.753 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:32.753 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:32.754 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:32.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2022-05-07T15:55:32.762 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:55:32.763 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:55:32.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:55:32.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:55:32.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:55:32.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:55:32.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.13828.out 2022-05-07T15:55:32.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.13828.out 2022-05-07T15:55:32.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2022-05-07T15:55:32.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2022-05-07T15:55:32.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2022-05-07T15:55:32.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2022-05-07T15:55:32.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2022-05-07T15:55:32.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.13828.out 2022-05-07T15:55:33.448 INFO:tasks.workunit.client.0.smithi059.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2022-05-07T15:55:33.465 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:023dd3d2:::existing_12:head# 2022-05-07T15:55:33.472 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:04c40a0c:::existing_93:head# 2022-05-07T15:55:33.484 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:064ff5fb:::existing_36:head# 2022-05-07T15:55:33.486 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:091f9f1d:::existing_94:head# 2022-05-07T15:55:33.486 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:0a865a93:::existing_54:head# 2022-05-07T15:55:33.494 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:0fed513c:::existing_41:head# 2022-05-07T15:55:33.494 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:13b9e0b8:::existing_92:head# 2022-05-07T15:55:33.500 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:169157fa:::existing_81:head# 2022-05-07T15:55:33.505 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:17731382:::existing_58:head# 2022-05-07T15:55:33.506 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:18a45a39:::existing_43:head# 2022-05-07T15:55:33.506 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:1c160f01:::existing_66:head# 2022-05-07T15:55:33.507 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:1dd6c249:::existing_97:head# 2022-05-07T15:55:33.508 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:227e0c77:::existing_7:head# 2022-05-07T15:55:33.508 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:23e1b063:::existing_59:head# 2022-05-07T15:55:33.509 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:28566edb:::existing_78:head# 2022-05-07T15:55:33.512 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:2a101182:::existing_20:head# 2022-05-07T15:55:33.522 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:2d1519da:::existing_96:head# 2022-05-07T15:55:33.522 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:3078b87a:::existing_86:head# 2022-05-07T15:55:33.523 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:30bde01b:::existing_15:head# 2022-05-07T15:55:33.525 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:317a7e19:::existing_56:head# 2022-05-07T15:55:33.525 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:38b9c7fa:::existing_33:head# 2022-05-07T15:55:33.535 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:3e16d84b:::existing_48:head# 2022-05-07T15:55:33.536 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:45fe8203:::existing_75:head# 2022-05-07T15:55:33.545 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:4e82c9ba:::existing_50:head# 2022-05-07T15:55:33.546 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5298dcdb:::existing_57:head# 2022-05-07T15:55:33.546 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:52ebaa44:::existing_32:head# 2022-05-07T15:55:33.551 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:53588912:::existing_9:head# 2022-05-07T15:55:33.552 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5b7e2de6:::existing_84:head# 2022-05-07T15:55:33.554 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5da0255c:::existing_13:head# 2022-05-07T15:55:33.554 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5da2dab8:::existing_53:head# 2022-05-07T15:55:33.555 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5ee3514b:::existing_30:head# 2022-05-07T15:55:33.559 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:602f83fe:::foo:head# 2022-05-07T15:55:33.559 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:60426cc7:::existing_62:head# 2022-05-07T15:55:33.560 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:628918ca:::existing_79:head# 2022-05-07T15:55:33.561 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:63f8ab2d:::existing_77:head# 2022-05-07T15:55:33.561 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:67e1780f:::existing_27:head# 2022-05-07T15:55:33.566 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:68143ca1:::existing_71:head# 2022-05-07T15:55:33.570 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:6bac9509:::existing_29:head# 2022-05-07T15:55:33.570 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:6bbec798:::existing_45:head# 2022-05-07T15:55:33.571 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:6cb23a40:::existing_65:head# 2022-05-07T15:55:33.571 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:7055ec5e:::existing_83:head# 2022-05-07T15:55:33.606 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:70a54e2b:::existing_87:head# 2022-05-07T15:55:33.614 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:750ace21:::existing_19:head# 2022-05-07T15:55:33.614 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:78623b1d:::existing_6:head# 2022-05-07T15:55:33.628 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:79881152:::existing_63:head# 2022-05-07T15:55:33.639 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:7a80c0d5:::existing_99:head# 2022-05-07T15:55:33.644 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:81490677:::existing_76:head# 2022-05-07T15:55:33.645 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:85ec8e5d:::existing_85:head# 2022-05-07T15:55:33.646 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:8e2abbf0:::existing_49:head# 2022-05-07T15:55:33.646 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:8ee0fe0f:::existing_52:head# 2022-05-07T15:55:33.647 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:923c5d4d:::existing_5:head# 2022-05-07T15:55:33.647 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:92731e71:::existing_89:head# 2022-05-07T15:55:33.648 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:928205cf:::existing_82:head# 2022-05-07T15:55:33.649 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:93be2267:::existing_24:head# 2022-05-07T15:55:33.649 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:99ceda2c:::existing_44:head# 2022-05-07T15:55:33.650 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:9dd917ee:::existing_26:head# 2022-05-07T15:55:33.651 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a28805a6:::existing_72:head# 2022-05-07T15:55:33.651 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a294a8f6:::existing_16:head# 2022-05-07T15:55:33.652 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a2f5473a:::existing_42:head# 2022-05-07T15:55:33.653 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a504a533:::existing_90:head# 2022-05-07T15:55:33.653 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a691337e:::existing_61:head# 2022-05-07T15:55:33.654 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a6e8d7ac:::existing_28:head# 2022-05-07T15:55:33.654 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a9fde86e:::existing_18:head# 2022-05-07T15:55:33.654 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:aa309faa:::existing_68:head# 2022-05-07T15:55:33.659 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:ae40b0ed:::existing_31:head# 2022-05-07T15:55:33.659 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:b279b1fe:::existing_10:head# 2022-05-07T15:55:33.660 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:b343e277:::existing_88:head# 2022-05-07T15:55:33.660 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:b7130e90:::existing_80:head# 2022-05-07T15:55:33.662 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:bc6f7309:::existing_70:head# 2022-05-07T15:55:33.663 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:be8489b7:::existing_38:head# 2022-05-07T15:55:33.664 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:beb4e5e3:::existing_47:head# 2022-05-07T15:55:33.665 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:bebec877:::existing_74:head# 2022-05-07T15:55:33.665 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c04347d8:::existing_60:head# 2022-05-07T15:55:33.666 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c156033e:::existing_100:head# 2022-05-07T15:55:33.666 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c19f5a8e:::existing_8:head# 2022-05-07T15:55:33.667 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c3e6ebe4:::existing_17:head# 2022-05-07T15:55:33.668 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c6c5a7c2:::existing_39:head# 2022-05-07T15:55:33.668 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:cc1d0abf:::existing_95:head# 2022-05-07T15:55:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:ccb3865b:::existing_73:head# 2022-05-07T15:55:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:cd84e19e:::existing_40:head# 2022-05-07T15:55:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d2cd64ce:::existing_37:head# 2022-05-07T15:55:33.670 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d41a25c9:::existing_22:head# 2022-05-07T15:55:33.671 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d7f088da:::existing_91:head# 2022-05-07T15:55:33.671 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d9cbd1d6:::existing_64:head# 2022-05-07T15:55:33.672 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:dcc4ff82:::existing_98:head# 2022-05-07T15:55:33.672 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:dd677bd3:::existing_35:head# 2022-05-07T15:55:33.672 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:dd6d90e7:::existing_46:head# 2022-05-07T15:55:33.673 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:e0e22077:::existing_21:head# 2022-05-07T15:55:33.674 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:e255b710:::existing_55:head# 2022-05-07T15:55:33.674 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:eff1d5db:::existing_14:head# 2022-05-07T15:55:33.674 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f1b156bf:::existing_23:head# 2022-05-07T15:55:33.675 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f5ce4199:::existing_34:head# 2022-05-07T15:55:33.676 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f6809456:::existing_11:head# 2022-05-07T15:55:33.676 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f699a2a9:::existing_67:head# 2022-05-07T15:55:33.681 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f9baac43:::existing_25:head# 2022-05-07T15:55:33.682 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:fb76688e:::existing_51:head# 2022-05-07T15:55:33.682 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:fbb92eec:::existing_69:head# 2022-05-07T15:55:33.683 INFO:tasks.workunit.client.0.smithi059.stderr:Export successful 2022-05-07T15:55:33.684 INFO:tasks.workunit.client.0.smithi059.stdout: marking collection for removal 2022-05-07T15:55:33.684 INFO:tasks.workunit.client.0.smithi059.stdout:setting '_remove' omap key 2022-05-07T15:55:33.684 INFO:tasks.workunit.client.0.smithi059.stdout:finish_remove_pgs 1.0_head removing 1.0 2022-05-07T15:55:33.856 INFO:tasks.workunit.client.0.smithi059.stderr:Remove successful 2022-05-07T15:55:34.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.13828.out 2022-05-07T15:55:34.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2022-05-07T15:55:34.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2022-05-07T15:55:34.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2022-05-07T15:55:34.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2022-05-07T15:55:34.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2022-05-07T15:55:34.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.13828.out 2022-05-07T15:55:34.819 INFO:tasks.workunit.client.0.smithi059.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2022-05-07T15:55:34.819 INFO:tasks.workunit.client.0.smithi059.stdout:Importing pgid 1.0 2022-05-07T15:55:34.820 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:023dd3d2:::existing_12:head# 2022-05-07T15:55:34.820 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:34.891 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:04c40a0c:::existing_93:head# 2022-05-07T15:55:34.891 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:34.924 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:064ff5fb:::existing_36:head# 2022-05-07T15:55:34.924 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:34.958 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:091f9f1d:::existing_94:head# 2022-05-07T15:55:34.958 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:34.991 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:0a865a93:::existing_54:head# 2022-05-07T15:55:34.992 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.025 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:0fed513c:::existing_41:head# 2022-05-07T15:55:35.025 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.058 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:13b9e0b8:::existing_92:head# 2022-05-07T15:55:35.058 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.091 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:169157fa:::existing_81:head# 2022-05-07T15:55:35.092 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.125 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:17731382:::existing_58:head# 2022-05-07T15:55:35.125 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.158 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:18a45a39:::existing_43:head# 2022-05-07T15:55:35.159 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.192 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:1c160f01:::existing_66:head# 2022-05-07T15:55:35.192 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.264 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:1dd6c249:::existing_97:head# 2022-05-07T15:55:35.265 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.300 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:227e0c77:::existing_7:head# 2022-05-07T15:55:35.301 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.334 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:23e1b063:::existing_59:head# 2022-05-07T15:55:35.334 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.367 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:28566edb:::existing_78:head# 2022-05-07T15:55:35.368 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.401 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:2a101182:::existing_20:head# 2022-05-07T15:55:35.401 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.434 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:2d1519da:::existing_96:head# 2022-05-07T15:55:35.435 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.468 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:3078b87a:::existing_86:head# 2022-05-07T15:55:35.468 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.501 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:30bde01b:::existing_15:head# 2022-05-07T15:55:35.501 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.535 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:317a7e19:::existing_56:head# 2022-05-07T15:55:35.535 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.568 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:38b9c7fa:::existing_33:head# 2022-05-07T15:55:35.568 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.629 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:3e16d84b:::existing_48:head# 2022-05-07T15:55:35.629 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.677 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:45fe8203:::existing_75:head# 2022-05-07T15:55:35.677 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.710 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:4e82c9ba:::existing_50:head# 2022-05-07T15:55:35.711 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.744 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5298dcdb:::existing_57:head# 2022-05-07T15:55:35.744 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.777 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:52ebaa44:::existing_32:head# 2022-05-07T15:55:35.778 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.844 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:53588912:::existing_9:head# 2022-05-07T15:55:35.844 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.878 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5b7e2de6:::existing_84:head# 2022-05-07T15:55:35.878 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.911 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5da0255c:::existing_13:head# 2022-05-07T15:55:35.911 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.944 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5da2dab8:::existing_53:head# 2022-05-07T15:55:35.945 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:35.978 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5ee3514b:::existing_30:head# 2022-05-07T15:55:35.979 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.020 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:602f83fe:::foo:head# 2022-05-07T15:55:36.020 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.063 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:60426cc7:::existing_62:head# 2022-05-07T15:55:36.063 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.103 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:628918ca:::existing_79:head# 2022-05-07T15:55:36.104 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.145 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:63f8ab2d:::existing_77:head# 2022-05-07T15:55:36.145 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.195 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:67e1780f:::existing_27:head# 2022-05-07T15:55:36.196 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.245 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:68143ca1:::existing_71:head# 2022-05-07T15:55:36.246 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.296 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:6bac9509:::existing_29:head# 2022-05-07T15:55:36.296 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.329 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:6bbec798:::existing_45:head# 2022-05-07T15:55:36.329 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.363 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:6cb23a40:::existing_65:head# 2022-05-07T15:55:36.363 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.421 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:7055ec5e:::existing_83:head# 2022-05-07T15:55:36.422 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.455 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:70a54e2b:::existing_87:head# 2022-05-07T15:55:36.455 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.488 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:750ace21:::existing_19:head# 2022-05-07T15:55:36.489 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.521 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:78623b1d:::existing_6:head# 2022-05-07T15:55:36.522 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.555 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:79881152:::existing_63:head# 2022-05-07T15:55:36.556 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.605 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:7a80c0d5:::existing_99:head# 2022-05-07T15:55:36.606 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.639 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:81490677:::existing_76:head# 2022-05-07T15:55:36.639 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.672 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:85ec8e5d:::existing_85:head# 2022-05-07T15:55:36.672 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.706 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:8e2abbf0:::existing_49:head# 2022-05-07T15:55:36.706 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.764 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:8ee0fe0f:::existing_52:head# 2022-05-07T15:55:36.765 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.789 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:923c5d4d:::existing_5:head# 2022-05-07T15:55:36.790 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:92731e71:::existing_89:head# 2022-05-07T15:55:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.839 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:928205cf:::existing_82:head# 2022-05-07T15:55:36.840 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.865 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:93be2267:::existing_24:head# 2022-05-07T15:55:36.866 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:99ceda2c:::existing_44:head# 2022-05-07T15:55:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.923 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:9dd917ee:::existing_26:head# 2022-05-07T15:55:36.924 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:36.982 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a28805a6:::existing_72:head# 2022-05-07T15:55:36.982 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.015 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a294a8f6:::existing_16:head# 2022-05-07T15:55:37.015 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.049 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a2f5473a:::existing_42:head# 2022-05-07T15:55:37.049 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.082 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a504a533:::existing_90:head# 2022-05-07T15:55:37.082 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.115 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a691337e:::existing_61:head# 2022-05-07T15:55:37.116 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.149 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a6e8d7ac:::existing_28:head# 2022-05-07T15:55:37.149 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.182 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a9fde86e:::existing_18:head# 2022-05-07T15:55:37.183 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.216 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:aa309faa:::existing_68:head# 2022-05-07T15:55:37.216 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.249 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:ae40b0ed:::existing_31:head# 2022-05-07T15:55:37.249 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.341 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:b279b1fe:::existing_10:head# 2022-05-07T15:55:37.341 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.374 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:b343e277:::existing_88:head# 2022-05-07T15:55:37.375 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.409 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:b7130e90:::existing_80:head# 2022-05-07T15:55:37.409 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.441 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:bc6f7309:::existing_70:head# 2022-05-07T15:55:37.442 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.475 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:be8489b7:::existing_38:head# 2022-05-07T15:55:37.475 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.542 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:beb4e5e3:::existing_47:head# 2022-05-07T15:55:37.542 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.575 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:bebec877:::existing_74:head# 2022-05-07T15:55:37.576 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.642 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c04347d8:::existing_60:head# 2022-05-07T15:55:37.642 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.675 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c156033e:::existing_100:head# 2022-05-07T15:55:37.676 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.709 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c19f5a8e:::existing_8:head# 2022-05-07T15:55:37.709 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.743 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c3e6ebe4:::existing_17:head# 2022-05-07T15:55:37.743 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.776 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c6c5a7c2:::existing_39:head# 2022-05-07T15:55:37.776 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.809 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:cc1d0abf:::existing_95:head# 2022-05-07T15:55:37.810 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.843 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:ccb3865b:::existing_73:head# 2022-05-07T15:55:37.843 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:cd84e19e:::existing_40:head# 2022-05-07T15:55:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.910 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d2cd64ce:::existing_37:head# 2022-05-07T15:55:37.910 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.943 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d41a25c9:::existing_22:head# 2022-05-07T15:55:37.947 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:37.977 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d7f088da:::existing_91:head# 2022-05-07T15:55:37.978 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.010 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d9cbd1d6:::existing_64:head# 2022-05-07T15:55:38.011 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.064 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:dcc4ff82:::existing_98:head# 2022-05-07T15:55:38.064 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.102 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:dd677bd3:::existing_35:head# 2022-05-07T15:55:38.102 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.135 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:dd6d90e7:::existing_46:head# 2022-05-07T15:55:38.136 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.169 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:e0e22077:::existing_21:head# 2022-05-07T15:55:38.169 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.202 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:e255b710:::existing_55:head# 2022-05-07T15:55:38.203 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.236 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:eff1d5db:::existing_14:head# 2022-05-07T15:55:38.237 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.270 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f1b156bf:::existing_23:head# 2022-05-07T15:55:38.270 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.303 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f5ce4199:::existing_34:head# 2022-05-07T15:55:38.304 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.336 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f6809456:::existing_11:head# 2022-05-07T15:55:38.337 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.370 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f699a2a9:::existing_67:head# 2022-05-07T15:55:38.370 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.403 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f9baac43:::existing_25:head# 2022-05-07T15:55:38.404 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.437 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:fb76688e:::existing_51:head# 2022-05-07T15:55:38.437 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.471 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:fbb92eec:::existing_69:head# 2022-05-07T15:55:38.472 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2022-05-07T15:55:38.503 INFO:tasks.workunit.client.0.smithi059.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2022-05-07T15:55:38.545 INFO:tasks.workunit.client.0.smithi059.stdout:Import successful 2022-05-07T15:55:39.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2022-05-07T15:55:39.098 INFO:tasks.workunit.client.0.smithi059.stdout:reviving divergent 1 2022-05-07T15:55:39.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:55:39.418 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:55:39.418 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 60778 0 0 16 16 peering 2022-05-07T15:55:30.640617+0000 29'106 46:117 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:55:39.419 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:55:39.419 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:55:39.420 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:55:39.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2022-05-07T15:55:39.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-05-07T15:55:39.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-05-07T15:55:39.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-05-07T15:55:39.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-05-07T15:55:39.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:55:39.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:55:39.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:55:39.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:55:39.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:55:39.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:55:39.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:55:39.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-05-07T15:55:39.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-05-07T15:55:39.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:55:39.434 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-05-07T15:55:39.434 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:55:39.434 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:55:39.436 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:55:39.436 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:55:39.437 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:55:39.437 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:55:39.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:55:39.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:55:39.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:55:39.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:55:39.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:55:39.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:55:39.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:55:39.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-05-07T15:55:39.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-05-07T15:55:39.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:55:39.440 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:55:39.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-05-07T15:55:39.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:55:39.442 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-05-07T15:55:39.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-05-07T15:55:39.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-05-07T15:55:39.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-05-07T15:55:39.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-05-07T15:55:39.470 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:39.471+0000 7f2059e92d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:55:39.474 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:39.475+0000 7f2059e92d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:55:39.475 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:39.479+0000 7f2059e92d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:55:39.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2022-05-07T15:55:39.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:55:39.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:55:39.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:55:39.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:55:39.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:55:39.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:55:39.782 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:55:39.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:55:39.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:55:40.005 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:40.007+0000 7f2059e92d80 -1 Falling back to public interface 2022-05-07T15:55:40.112 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/29376,v1:127.0.0.1:6819/29376] [v2:127.0.0.1:6820/29376,v1:127.0.0.1:6821/29376] exists,up 389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:55:40.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:55:40.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:55:40.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:55:40.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2022-05-07T15:55:40.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:55:40.115 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:55:40.115 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:55:40.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:55:40.117 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:55:40.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:55:40.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:55:40.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:55:40.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:55:40.450 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/29376,v1:127.0.0.1:6819/29376] [v2:127.0.0.1:6820/29376,v1:127.0.0.1:6821/29376] exists,up 389e3986-a3c9-4ea7-b566-1125c8d4e23a 2022-05-07T15:55:40.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:55:40.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:55:40.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:55:40.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2022-05-07T15:55:40.870 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:55:40.871+0000 7f2059e92d80 -1 osd.1 47 log_to_monitors {default=true} 2022-05-07T15:56:00.455 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2022-05-07T15:56:00.456 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-05-07T15:56:00.456 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-05-07T15:56:00.456 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:56:00.457 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:00.457 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:00.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.1.asok 2022-05-07T15:56:00.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2022-05-07T15:56:00.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.1.asok dump_ops_in_flight 2022-05-07T15:56:00.542 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:56:00.542 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [], 2022-05-07T15:56:00.543 INFO:tasks.workunit.client.0.smithi059.stdout: "num_ops": 0 2022-05-07T15:56:00.543 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:56:00.550 INFO:tasks.workunit.client.0.smithi059.stdout:allowing recovery 2022-05-07T15:56:00.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2022-05-07T15:56:00.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:56:00.860 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:56:00.861 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 62473 0 0 12 12 active+clean 2022-05-07T15:55:42.672671+0000 47'102 51:147 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:56:00.861 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:56:00.861 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:56:00.863 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:56:00.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2022-05-07T15:56:00.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2022-05-07T15:56:01.007 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-05-07T15:56:01.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2022-05-07T15:56:01.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2022-05-07T15:56:01.153 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-05-07T15:56:01.160 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2022-05-07T15:56:01.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2022-05-07T15:56:01.301 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-05-07T15:56:01.308 INFO:tasks.workunit.client.0.smithi059.stdout:reading divergent objects 2022-05-07T15:56:01.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2022-05-07T15:56:01.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2022-05-07T15:56:01.613 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:56:01.613 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 62473 0 0 12 12 active+clean 2022-05-07T15:55:42.672671+0000 47'102 51:147 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:53:25.378865+0000 0'0 2022-05-07T15:53:25.378865+0000 0 2022-05-07T15:56:01.614 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:56:01.614 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:56:01.616 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:56:01.625 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2022-05-07T15:56:01.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2022-05-07T15:56:01.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:56:01.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2022-05-07T15:56:01.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:56:01.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2022-05-07T15:56:01.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:56:01.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2022-05-07T15:56:01.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-05-07T15:56:01.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2022-05-07T15:56:01.796 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2022-05-07T15:56:01.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-05-07T15:56:01.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2022-05-07T15:56:01.836 INFO:tasks.workunit.client.0.smithi059.stderr:error getting test/newobject_1: (2) No such file or directory 2022-05-07T15:56:01.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-05-07T15:56:01.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2022-05-07T15:56:01.877 INFO:tasks.workunit.client.0.smithi059.stderr:error getting test/newobject_2: (2) No such file or directory 2022-05-07T15:56:01.880 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2022-05-07T15:56:01.882 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2022-05-07T15:56:01.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2022-05-07T15:56:01.890 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'102 (29'4) delete 1:70b3bf12:::existing_4:head by client.4533.0:1 2022-05-07T15:53:55.898173+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2022-05-07T15:56:01.891 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: keeping 29'102 (29'4) delete 1:70b3bf12:::existing_4:head by client.4533.0:1 2022-05-07T15:53:55.898173+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2022-05-07T15:56:01.891 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-05-07T15:56:01.891 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'102 last_divergent_update: 29'102 2022-05-07T15:56:01.892 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4654.0:1 2022-05-07T15:54:48.749111+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0, already merged 2022-05-07T15:56:01.892 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2022-05-07T15:53:55.895541+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.892 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2022-05-07T15:53:55.895541+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.892 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-05-07T15:56:01.892 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2022-05-07T15:56:01.893 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2022-05-07T15:56:01.893 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2022-05-07T15:56:01.893 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4544.0:1 2022-05-07T15:53:55.907722+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2022-05-07T15:56:01.893 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4544.0:1 2022-05-07T15:53:55.907722+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2022-05-07T15:56:01.894 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2022-05-07T15:56:01.894 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2022-05-07T15:56:01.894 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2022-05-07T15:56:01.894 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2022-05-07T15:56:01.895 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2022-05-07T15:56:01.895 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4544.0:1 2022-05-07T15:53:55.907722+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2022-05-07T15:56:01.895 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2022-05-07T15:56:01.896 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2022-05-07T15:56:01.896 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'106 (29'2) modify 1:97db2a0f:::existing_2:head by client.4545.0:1 2022-05-07T15:53:55.908191+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.896 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: keeping 29'106 (29'2) modify 1:97db2a0f:::existing_2:head by client.4545.0:1 2022-05-07T15:53:55.908191+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.896 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-05-07T15:56:01.897 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'106 last_divergent_update: 29'106 2022-05-07T15:56:01.897 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2022-05-07T15:56:01.897 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2022-05-07T15:56:01.897 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2022-05-07T15:56:01.898 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'106 (29'2) modify 1:97db2a0f:::existing_2:head by client.4545.0:1 2022-05-07T15:53:55.908191+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.898 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2022-05-07T15:56:01.898 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2022-05-07T15:56:01.898 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4543.0:1 2022-05-07T15:53:55.906372+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.899 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4543.0:1 2022-05-07T15:53:55.906372+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.899 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-05-07T15:56:01.899 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2022-05-07T15:56:01.899 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2022-05-07T15:56:01.899 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2022-05-07T15:56:01.900 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2022-05-07T15:56:01.900 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4543.0:1 2022-05-07T15:53:55.906372+0000 0 ObjectCleanRegions clean_offsets: [773~18446744073709550842], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.900 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2022-05-07T15:56:01.900 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2022-05-07T15:56:01.901 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4542.0:1 2022-05-07T15:53:55.905775+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.901 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4542.0:1 2022-05-07T15:53:55.905775+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-05-07T15:56:01.901 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-05-07T15:56:01.901 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2022-05-07T15:56:01.902 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2022-05-07T15:56:01.902 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2022-05-07T15:55:30.831+0000 7f29cd629700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2022-05-07T15:56:01.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2022-05-07T15:56:01.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2022-05-07T15:56:01.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2022-05-07T15:56:01.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2022-05-07T15:56:01.907 INFO:tasks.workunit.client.0.smithi059.stdout:success 2022-05-07T15:56:01.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2022-05-07T15:56:01.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.13828.out 2022-05-07T15:56:01.909 INFO:tasks.workunit.client.0.smithi059.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.13828.out': No such file or directory 2022-05-07T15:56:01.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2022-05-07T15:56:01.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2022-05-07T15:56:01.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2022-05-07T15:56:02.374 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' does not exist 2022-05-07T15:56:02.382 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2022-05-07T15:56:02.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:56:02.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:56:02.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:56:02.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:56:02.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:56:02.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:56:02.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2022-05-07T15:56:02.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-05-07T15:56:02.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:56:02.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-05-07T15:56:02.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:56:02.504 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:56:02.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:56:02.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:56:02.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:56:02.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:56:02.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:56:02.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:56:02.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:56:02.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:56:02.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:56:02.523 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:56:02.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:56:02.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:56:02.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:56:02.525 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:56:02.525 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:56:02.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:56:02.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-05-07T15:56:02.541 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:56:02.542 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:02.542 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:02.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13828 2022-05-07T15:56:02.545 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:56:02.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:56:02.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2022-05-07T15:56:02.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2022-05-07T15:56:02.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2022-05-07T15:56:02.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2022-05-07T15:56:02.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-05-07T15:56:02.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-05-07T15:56:02.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-05-07T15:56:02.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T15:56:02.550 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T15:56:02.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T15:56:02.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T15:56:02.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T15:56:02.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T15:56:02.554 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T15:56:02.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T15:56:02.555 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T15:56:02.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T15:56:02.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T15:56:02.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T15:56:02.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:56:02.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T15:56:02.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T15:56:02.559 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T15:56:02.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T15:56:02.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-05-07T15:56:02.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-05-07T15:56:02.563 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T15:56:02.563 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:02.563 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:02.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13828 2022-05-07T15:56:02.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-05-07T15:56:02.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-05-07T15:56:02.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2022-05-07T15:56:02.567 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-05-07T15:56:02.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:02.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:02.569 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.13828 2022-05-07T15:56:02.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-05-07T15:56:02.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2022-05-07T15:56:02.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-05-07T15:56:02.573 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2022-05-07T15:56:02.573 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2022-05-07T15:56:02.573 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2022-05-07T15:56:02.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2022-05-07T15:56:02.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2022-05-07T15:56:02.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2022-05-07T15:56:02.575 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2022-05-07T15:56:02.577 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2022-05-07T15:56:02.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2022-05-07T15:56:02.577 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:56:02.578 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:56:02.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2022-05-07T15:56:02.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2022-05-07T15:56:02.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-05-07T15:56:02.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-05-07T15:56:02.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-05-07T15:56:02.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2022-05-07T15:56:02.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2022-05-07T15:56:02.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-05-07T15:56:02.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:56:02.904 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:56:02.905 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:56:02.905 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:02.905 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:02.905 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:02.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-05-07T15:56:03.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-05-07T15:56:03.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-05-07T15:56:03.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:56:03.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:56:03.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-05-07T15:56:03.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-05-07T15:56:03.098 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:56:03.098 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:56:03.099 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:56:03.100 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:56:03.100 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:03.100 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:03.101 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-mon.a.asok 2022-05-07T15:56:03.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:56:03.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.13828/ceph-mon.a.asok config get fsid 2022-05-07T15:56:03.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-05-07T15:56:03.187 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-05-07T15:56:03.187 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-05-07T15:56:03.187 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-05-07T15:56:03.188 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-05-07T15:56:03.189 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-05-07T15:56:03.189 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-05-07T15:56:03.189 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-05-07T15:56:03.189 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:56:03.190 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:03.190 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:03.190 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-mon.a.asok 2022-05-07T15:56:03.190 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-05-07T15:56:03.191 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.13828/ceph-mon.a.asok config get mon_host 2022-05-07T15:56:03.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2022-05-07T15:56:03.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2022-05-07T15:56:03.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-05-07T15:56:03.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-05-07T15:56:03.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-05-07T15:56:03.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2022-05-07T15:56:03.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-05-07T15:56:04.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-05-07T15:56:04.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:56:04.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:56:04.164 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:56:04.164 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:04.164 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:04.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:04.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:56:04.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-05-07T15:56:04.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2022-05-07T15:56:04.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2022-05-07T15:56:04.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:56:04.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:56:04.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-05-07T15:56:04.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:56:04.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2022-05-07T15:56:04.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:56:04.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:56:04.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:56:04.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:56:04.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-05-07T15:56:04.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-05-07T15:56:04.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:56:04.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:56:04.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:56:04.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:56:04.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:56:04.205 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:56:04.205 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:56:04.205 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:04.205 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:04.205 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:04.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:04.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:56:04.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:56:04.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:56:04.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:56:04.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:56:04.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:56:04.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:56:04.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:56:04.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:56:04.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:56:04.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2022-05-07T15:56:04.210 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:56:04.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=77f94f7d-23a4-46e9-b8c3-ecb7deb5d3f3 2022-05-07T15:56:04.211 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 77f94f7d-23a4-46e9-b8c3-ecb7deb5d3f3 2022-05-07T15:56:04.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 77f94f7d-23a4-46e9-b8c3-ecb7deb5d3f3' 2022-05-07T15:56:04.211 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:56:04.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCUlnZiYjeEDhAABCuj1v+uvnzCZhGYO87qHQ== 2022-05-07T15:56:04.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCUlnZiYjeEDhAABCuj1v+uvnzCZhGYO87qHQ=="}' 2022-05-07T15:56:04.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 77f94f7d-23a4-46e9-b8c3-ecb7deb5d3f3 -i td/divergent-priors/0/new.json 2022-05-07T15:56:04.526 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:56:04.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2022-05-07T15:56:04.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCUlnZiYjeEDhAABCuj1v+uvnzCZhGYO87qHQ== --osd-uuid 77f94f7d-23a4-46e9-b8c3-ecb7deb5d3f3 2022-05-07T15:56:04.574 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.575+0000 7f79f2f4fd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:04.576 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.579+0000 7f79f2f4fd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:04.578 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.579+0000 7f79f2f4fd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:04.579 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.583+0000 7f79f2f4fd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:56:04.579 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.583+0000 7f79f2f4fd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:56:04.579 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.583+0000 7f79f2f4fd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-05-07T15:56:04.615 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:04.619+0000 7f79f2f4fd80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2022-05-07T15:56:05.026 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:05.027+0000 7f79f2f4fd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:56:07.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2022-05-07T15:56:07.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:56:07.729 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2022-05-07T15:56:07.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-05-07T15:56:07.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:56:08.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-05-07T15:56:08.172 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2022-05-07T15:56:08.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:56:08.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:56:08.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:56:08.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:56:08.208 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:08.211+0000 7f339e8a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:08.210 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:08.211+0000 7f339e8a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:08.211 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:08.215+0000 7f339e8a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:08.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-05-07T15:56:08.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:56:08.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-05-07T15:56:08.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:56:08.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:56:08.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:08.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:56:08.500 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:56:08.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:08.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:56:08.753 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:08.755+0000 7f339e8a1d80 -1 Falling back to public interface 2022-05-07T15:56:08.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:56:09.604 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:09.607+0000 7f339e8a1d80 -1 osd.0 0 log_to_monitors {default=true} 2022-05-07T15:56:09.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:56:09.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:09.832 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:56:09.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:56:09.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:09.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:56:10.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:56:11.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:56:11.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:11.229 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:56:11.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:56:11.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:11.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-05-07T15:56:11.577 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30845,v1:127.0.0.1:6803/30845] [v2:127.0.0.1:6804/30845,v1:127.0.0.1:6805/30845] exists,up 77f94f7d-23a4-46e9-b8c3-ecb7deb5d3f3 2022-05-07T15:56:11.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:56:11.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:56:11.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:56:11.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2022-05-07T15:56:11.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2022-05-07T15:56:11.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:56:11.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:56:11.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-05-07T15:56:11.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:56:11.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2022-05-07T15:56:11.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:56:11.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:56:11.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:56:11.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:56:11.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-05-07T15:56:11.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-05-07T15:56:11.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:56:11.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:56:11.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:56:11.586 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:56:11.586 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:56:11.586 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:56:11.586 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:56:11.586 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:11.587 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:11.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:11.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:11.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:56:11.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:56:11.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:56:11.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:56:11.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:56:11.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:56:11.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:56:11.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:56:11.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:56:11.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:56:11.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2022-05-07T15:56:11.590 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:56:11.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f155a23c-aa9c-4517-acb1-2eba27c1dc04 2022-05-07T15:56:11.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 f155a23c-aa9c-4517-acb1-2eba27c1dc04' 2022-05-07T15:56:11.591 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 f155a23c-aa9c-4517-acb1-2eba27c1dc04 2022-05-07T15:56:11.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:56:11.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCblnZiXEfYJBAAzI+MxbiYiDJJeNRCxCuLpw== 2022-05-07T15:56:11.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCblnZiXEfYJBAAzI+MxbiYiDJJeNRCxCuLpw=="}' 2022-05-07T15:56:11.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f155a23c-aa9c-4517-acb1-2eba27c1dc04 -i td/divergent-priors/1/new.json 2022-05-07T15:56:12.059 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:56:12.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2022-05-07T15:56:12.071 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCblnZiXEfYJBAAzI+MxbiYiDJJeNRCxCuLpw== --osd-uuid f155a23c-aa9c-4517-acb1-2eba27c1dc04 2022-05-07T15:56:12.110 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.111+0000 7f50f10b6d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:12.112 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.115+0000 7f50f10b6d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:12.114 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.115+0000 7f50f10b6d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:12.115 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.119+0000 7f50f10b6d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:56:12.115 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.119+0000 7f50f10b6d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:56:12.116 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.119+0000 7f50f10b6d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-05-07T15:56:12.148 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.151+0000 7f50f10b6d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2022-05-07T15:56:12.550 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:12.555+0000 7f50f10b6d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:56:15.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2022-05-07T15:56:15.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:56:15.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-05-07T15:56:15.469 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2022-05-07T15:56:15.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:56:15.902 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2022-05-07T15:56:15.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-05-07T15:56:15.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:56:15.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:56:15.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:56:15.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:56:15.933 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:15.935+0000 7f3bdecedd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:15.935 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:15.939+0000 7f3bdecedd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:15.937 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:15.939+0000 7f3bdecedd80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:16.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-05-07T15:56:16.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:56:16.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-05-07T15:56:16.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:56:16.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:56:16.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:16.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:56:16.259 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:56:16.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:16.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:56:16.472 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:16.475+0000 7f3bdecedd80 -1 Falling back to public interface 2022-05-07T15:56:16.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:56:17.238 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:17.239+0000 7f3bdecedd80 -1 osd.1 0 log_to_monitors {default=true} 2022-05-07T15:56:17.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:56:17.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:17.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:56:17.566 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:56:17.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:17.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-05-07T15:56:17.945 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/31228,v1:127.0.0.1:6811/31228] [v2:127.0.0.1:6812/31228,v1:127.0.0.1:6813/31228] exists,up f155a23c-aa9c-4517-acb1-2eba27c1dc04 2022-05-07T15:56:17.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:56:17.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:56:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:56:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2022-05-07T15:56:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2022-05-07T15:56:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-05-07T15:56:17.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-05-07T15:56:17.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-05-07T15:56:17.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-05-07T15:56:17.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2022-05-07T15:56:17.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-05-07T15:56:17.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-05-07T15:56:17.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-05-07T15:56:17.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-05-07T15:56:17.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-05-07T15:56:17.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-05-07T15:56:17.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-05-07T15:56:17.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-05-07T15:56:17.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-05-07T15:56:17.953 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-05-07T15:56:17.953 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-05-07T15:56:17.953 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-05-07T15:56:17.953 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-05-07T15:56:17.953 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:17.955 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:17.955 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:17.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' 2022-05-07T15:56:17.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-05-07T15:56:17.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-05-07T15:56:17.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-05-07T15:56:17.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-05-07T15:56:17.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-05-07T15:56:17.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-05-07T15:56:17.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-05-07T15:56:17.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:56:17.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-05-07T15:56:17.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-05-07T15:56:17.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2022-05-07T15:56:17.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-05-07T15:56:17.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=513e8827-8e2b-46d4-b2d6-3fe58dff16c2 2022-05-07T15:56:17.960 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 513e8827-8e2b-46d4-b2d6-3fe58dff16c2 2022-05-07T15:56:17.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 513e8827-8e2b-46d4-b2d6-3fe58dff16c2' 2022-05-07T15:56:17.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-05-07T15:56:17.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQChlnZisNXBOhAAinyGaWMYxl/Rt8DvGs+23Q== 2022-05-07T15:56:17.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQChlnZisNXBOhAAinyGaWMYxl/Rt8DvGs+23Q=="}' 2022-05-07T15:56:17.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 513e8827-8e2b-46d4-b2d6-3fe58dff16c2 -i td/divergent-priors/2/new.json 2022-05-07T15:56:18.504 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:56:18.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2022-05-07T15:56:18.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQChlnZisNXBOhAAinyGaWMYxl/Rt8DvGs+23Q== --osd-uuid 513e8827-8e2b-46d4-b2d6-3fe58dff16c2 2022-05-07T15:56:18.555 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.559+0000 7fb064132d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:18.558 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.563+0000 7fb064132d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:18.560 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.563+0000 7fb064132d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:18.561 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.563+0000 7fb064132d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:56:18.561 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.563+0000 7fb064132d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:56:18.561 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.563+0000 7fb064132d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-05-07T15:56:18.599 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:18.603+0000 7fb064132d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2022-05-07T15:56:19.042 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:19.047+0000 7fb064132d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-05-07T15:56:21.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2022-05-07T15:56:21.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-05-07T15:56:21.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-05-07T15:56:21.597 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2022-05-07T15:56:21.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-05-07T15:56:22.024 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2022-05-07T15:56:22.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-05-07T15:56:22.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=c67bfd60-15c3-4eb2-a02f-223bbda3f2bb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.13828/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-05-07T15:56:22.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-05-07T15:56:22.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-05-07T15:56:22.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-05-07T15:56:22.061 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:22.063+0000 7f8134267d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:22.064 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:22.067+0000 7f8134267d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:22.066 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:22.067+0000 7f8134267d80 -1 WARNING: all dangerous and experimental features are enabled. 2022-05-07T15:56:22.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-05-07T15:56:22.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-05-07T15:56:22.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-05-07T15:56:22.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-05-07T15:56:22.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-05-07T15:56:22.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:22.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-05-07T15:56:22.376 INFO:tasks.workunit.client.0.smithi059.stdout:0 2022-05-07T15:56:22.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:22.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:56:22.600 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:22.603+0000 7f8134267d80 -1 Falling back to public interface 2022-05-07T15:56:22.705 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:56:23.471 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T15:56:23.475+0000 7f8134267d80 -1 osd.2 0 log_to_monitors {default=true} 2022-05-07T15:56:23.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:56:23.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:23.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-05-07T15:56:23.708 INFO:tasks.workunit.client.0.smithi059.stdout:1 2022-05-07T15:56:23.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:23.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:56:24.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-05-07T15:56:25.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-05-07T15:56:25.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-05-07T15:56:25.112 INFO:tasks.workunit.client.0.smithi059.stdout:2 2022-05-07T15:56:25.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-05-07T15:56:25.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-05-07T15:56:25.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-05-07T15:56:25.468 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31583,v1:127.0.0.1:6819/31583] [v2:127.0.0.1:6820/31583,v1:127.0.0.1:6821/31583] exists,up 513e8827-8e2b-46d4-b2d6-3fe58dff16c2 2022-05-07T15:56:25.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-05-07T15:56:25.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-05-07T15:56:25.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-05-07T15:56:25.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2022-05-07T15:56:26.066 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2022-05-07T15:56:26.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2022-05-07T15:56:26.876 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2022-05-07T15:56:26.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2022-05-07T15:56:27.697 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2022-05-07T15:56:27.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2022-05-07T15:56:27.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2022-05-07T15:56:28.246 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2022-05-07T15:56:28.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-05-07T15:56:29.261 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2022-05-07T15:56:29.944 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 3 2022-05-07T15:56:29.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2022-05-07T15:56:30.763 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2022-05-07T15:56:30.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2022-05-07T15:56:31.539 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 pg_autoscale_mode to on 2022-05-07T15:56:31.552 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2022-05-07T15:56:31.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:56:31.553 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:56:31.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:56:31.897 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:56:31.897 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:56:31.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:56:31.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:31.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:56:32.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2022-05-07T15:56:32.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2022-05-07T15:56:32.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2022-05-07T15:56:32.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:32.056 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:56:32.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2022-05-07T15:56:32.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2022-05-07T15:56:32.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2022-05-07T15:56:32.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:32.200 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:56:32.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542148 2022-05-07T15:56:32.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542148 2022-05-07T15:56:32.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-60129542148' 2022-05-07T15:56:32.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:32.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2022-05-07T15:56:32.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:32.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:56:32.354 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2022-05-07T15:56:32.354 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:32.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2022-05-07T15:56:32.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2022-05-07T15:56:32.357 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836487 2022-05-07T15:56:32.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:56:32.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2022-05-07T15:56:32.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:56:33.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:56:33.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:56:34.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2022-05-07T15:56:34.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:34.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2022-05-07T15:56:34.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:34.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:56:34.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2022-05-07T15:56:34.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:34.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2022-05-07T15:56:34.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2022-05-07T15:56:34.028 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2022-05-07T15:56:34.028 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:56:34.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2022-05-07T15:56:34.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:34.360 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542148 2022-05-07T15:56:34.361 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:34.362 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:56:34.364 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2022-05-07T15:56:34.364 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:34.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542148 2022-05-07T15:56:34.366 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542148 2022-05-07T15:56:34.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2022-05-07T15:56:34.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:56:34.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542148 -lt 60129542148 2022-05-07T15:56:34.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2022-05-07T15:56:34.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:56:34.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:56:34.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:56:34.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:56:34.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2022-05-07T15:56:34.734 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:56:34.734 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:56:34.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:56:34.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:56:34.735 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:56:34.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:56:35.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:56:35.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:56:35.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:56:35.056 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:56:35.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:56:35.301 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:56:35.301 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:56:35.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:56:35.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:35.302 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:56:35.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2022-05-07T15:56:35.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2022-05-07T15:56:35.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2022-05-07T15:56:35.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:35.458 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:56:35.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2022-05-07T15:56:35.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2022-05-07T15:56:35.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2022-05-07T15:56:35.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:35.610 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:56:35.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542151 2022-05-07T15:56:35.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542151 2022-05-07T15:56:35.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151' 2022-05-07T15:56:35.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:35.762 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2022-05-07T15:56:35.762 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:35.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:56:35.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2022-05-07T15:56:35.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:35.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2022-05-07T15:56:35.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2022-05-07T15:56:35.768 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836489 2022-05-07T15:56:35.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:56:36.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2022-05-07T15:56:36.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:56:37.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:56:37.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:56:37.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836489 2022-05-07T15:56:37.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:37.467 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2022-05-07T15:56:37.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:37.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:56:37.471 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2022-05-07T15:56:37.471 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:37.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2022-05-07T15:56:37.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2022-05-07T15:56:37.474 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672968 2022-05-07T15:56:37.474 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:56:37.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2022-05-07T15:56:37.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:37.793 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542151 2022-05-07T15:56:37.793 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:37.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:56:37.797 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542151 2022-05-07T15:56:37.797 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:37.799 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542151 2022-05-07T15:56:37.799 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2022-05-07T15:56:37.800 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542151 2022-05-07T15:56:37.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:56:38.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542151 -lt 60129542151 2022-05-07T15:56:38.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:56:38.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:56:38.129 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:56:38.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:56:38.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:56:38.537 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:56:38.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:56:38.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:56:38.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:56:38.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:56:38.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:56:38.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-05-07T15:56:38.868 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:56:38.869 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:56:38.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:56:39.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-05-07T15:56:39.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-05-07T15:56:39.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-05-07T15:56:39.276 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2022-05-07T15:56:39.277 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2022-05-07T15:56:39.590 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:56:39.599 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2022-05-07T15:56:39.599 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2022-05-07T15:56:39.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2022-05-07T15:56:39.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2022-05-07T15:56:39.896 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:56:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-05-07T15:56:30.020382+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:56:28.061611+0000 0'0 2022-05-07T15:56:28.061611+0000 0 2022-05-07T15:56:39.897 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:56:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:56:39.904 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:56:39.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2022-05-07T15:56:39.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2022-05-07T15:56:39.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2022-05-07T15:56:39.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2022-05-07T15:56:39.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2022-05-07T15:56:39.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2022-05-07T15:56:39.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2022-05-07T15:56:39.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2022-05-07T15:56:39.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2022-05-07T15:56:39.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2022-05-07T15:56:39.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2022-05-07T15:56:39.910 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2022-05-07T15:56:39.911 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2022-05-07T15:56:39.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:39.912 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2022-05-07T15:56:40.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2022-05-07T15:56:40.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2022-05-07T15:56:40.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2022-05-07T15:56:40.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2022-05-07T15:56:40.645 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2022-05-07T15:56:40.821 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.821 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2022-05-07T15:56:40.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:40.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2022-05-07T15:56:41.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:41.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2022-05-07T15:56:41.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:41.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2022-05-07T15:56:41.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:41.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2022-05-07T15:56:41.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:41.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2022-05-07T15:56:41.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:41.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2022-05-07T15:56:41.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:41.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2022-05-07T15:56:42.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2022-05-07T15:56:42.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2022-05-07T15:56:42.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2022-05-07T15:56:42.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2022-05-07T15:56:42.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2022-05-07T15:56:42.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2022-05-07T15:56:42.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:42.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2022-05-07T15:56:43.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:43.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2022-05-07T15:56:43.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:43.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2022-05-07T15:56:43.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:43.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2022-05-07T15:56:43.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:43.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2022-05-07T15:56:43.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:43.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2022-05-07T15:56:43.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:43.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2022-05-07T15:56:44.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2022-05-07T15:56:44.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2022-05-07T15:56:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2022-05-07T15:56:44.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2022-05-07T15:56:44.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2022-05-07T15:56:44.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2022-05-07T15:56:44.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:44.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2022-05-07T15:56:45.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:45.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2022-05-07T15:56:45.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:45.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2022-05-07T15:56:45.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:45.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2022-05-07T15:56:45.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:45.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2022-05-07T15:56:45.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:45.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2022-05-07T15:56:45.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:45.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2022-05-07T15:56:46.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.008 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2022-05-07T15:56:46.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2022-05-07T15:56:46.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2022-05-07T15:56:46.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2022-05-07T15:56:46.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2022-05-07T15:56:46.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2022-05-07T15:56:46.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:46.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2022-05-07T15:56:47.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2022-05-07T15:56:47.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2022-05-07T15:56:47.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2022-05-07T15:56:47.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2022-05-07T15:56:47.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2022-05-07T15:56:47.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2022-05-07T15:56:47.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:47.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2022-05-07T15:56:48.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2022-05-07T15:56:48.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2022-05-07T15:56:48.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2022-05-07T15:56:48.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2022-05-07T15:56:48.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2022-05-07T15:56:48.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2022-05-07T15:56:48.943 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:48.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2022-05-07T15:56:49.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2022-05-07T15:56:49.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2022-05-07T15:56:49.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2022-05-07T15:56:49.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2022-05-07T15:56:49.694 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2022-05-07T15:56:49.820 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.821 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2022-05-07T15:56:49.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:49.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2022-05-07T15:56:50.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:50.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2022-05-07T15:56:50.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:50.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2022-05-07T15:56:50.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:50.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2022-05-07T15:56:50.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:50.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2022-05-07T15:56:50.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:50.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2022-05-07T15:56:50.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:50.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2022-05-07T15:56:51.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2022-05-07T15:56:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2022-05-07T15:56:51.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2022-05-07T15:56:51.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2022-05-07T15:56:51.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2022-05-07T15:56:51.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2022-05-07T15:56:51.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2022-05-07T15:56:51.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:51.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2022-05-07T15:56:52.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2022-05-07T15:56:52.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2022-05-07T15:56:52.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2022-05-07T15:56:52.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2022-05-07T15:56:52.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2022-05-07T15:56:52.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2022-05-07T15:56:52.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:52.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2022-05-07T15:56:53.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:53.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2022-05-07T15:56:53.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:53.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2022-05-07T15:56:53.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:53.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2022-05-07T15:56:53.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:53.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2022-05-07T15:56:53.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:53.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2022-05-07T15:56:53.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:53.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2022-05-07T15:56:54.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:54.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2022-05-07T15:56:54.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:54.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2022-05-07T15:56:54.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:54.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2022-05-07T15:56:54.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:54.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2022-05-07T15:56:54.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-05-07T15:56:54.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2022-05-07T15:56:54.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2022-05-07T15:56:54.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2022-05-07T15:56:54.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-05-07T15:56:54.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-05-07T15:56:54.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-05-07T15:56:54.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-05-07T15:56:54.838 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-05-07T15:56:54.838 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-05-07T15:56:54.838 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-05-07T15:56:54.839 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-05-07T15:56:54.839 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-05-07T15:56:54.839 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-05-07T15:56:54.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-05-07T15:56:54.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-05-07T15:56:54.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-05-07T15:56:54.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-05-07T15:56:54.980 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-05-07T15:56:55.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-05-07T15:56:55.319 INFO:tasks.workunit.client.0.smithi059.stderr:1 2022-05-07T15:56:55.320 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2022-05-07T15:56:55.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-05-07T15:56:55.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:55.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-05-07T15:56:55.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2022-05-07T15:56:55.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2022-05-07T15:56:55.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2022-05-07T15:56:55.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:55.477 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-05-07T15:56:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2022-05-07T15:56:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2022-05-07T15:56:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2022-05-07T15:56:55.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-05-07T15:56:55.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-05-07T15:56:55.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542157 2022-05-07T15:56:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542157 2022-05-07T15:56:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-60129542157' 2022-05-07T15:56:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:55.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2022-05-07T15:56:55.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:55.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-05-07T15:56:55.779 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2022-05-07T15:56:55.779 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:55.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2022-05-07T15:56:55.780 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836495 2022-05-07T15:56:55.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2022-05-07T15:56:55.781 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:56:56.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836495 2022-05-07T15:56:56.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-05-07T15:56:57.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-05-07T15:56:57.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-05-07T15:56:57.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836495 2022-05-07T15:56:57.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:57.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2022-05-07T15:56:57.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:57.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-05-07T15:56:57.500 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2022-05-07T15:56:57.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:57.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2022-05-07T15:56:57.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2022-05-07T15:56:57.503 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672974 2022-05-07T15:56:57.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-05-07T15:56:57.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2022-05-07T15:56:57.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-05-07T15:56:57.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542157 2022-05-07T15:56:57.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-05-07T15:56:57.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-05-07T15:56:57.874 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542157 2022-05-07T15:56:57.874 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-05-07T15:56:57.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542157 2022-05-07T15:56:57.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2022-05-07T15:56:57.876 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542157 2022-05-07T15:56:57.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-05-07T15:56:58.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542157 -lt 60129542157 2022-05-07T15:56:58.207 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-05-07T15:56:58.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:56:58.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:56:58.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-05-07T15:56:58.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-05-07T15:56:58.640 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-05-07T15:56:58.641 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-05-07T15:56:58.641 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-05-07T15:56:58.641 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-05-07T15:56:58.641 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-05-07T15:56:58.642 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-05-07T15:56:58.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-05-07T15:56:58.957 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-05-07T15:56:58.958 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-05-07T15:56:58.958 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-05-07T15:56:59.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-05-07T15:56:59.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-05-07T15:56:59.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-05-07T15:56:59.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2022-05-07T15:56:59.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2022-05-07T15:56:59.349 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2022-05-07T15:56:59.349 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2022-05-07T15:56:59.349 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2022-05-07T15:56:59.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2022-05-07T15:56:59.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2022-05-07T15:56:59.681 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2022-05-07T15:56:59.681 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2022-05-07T15:56:59.982 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-05-07T15:56:59.982 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2022-05-07T15:56:30.020382+0000 30'100 30:113 [1,0,2] 1 [1,0,2] 1 0'0 2022-05-07T15:56:28.061611+0000 0'0 2022-05-07T15:56:28.061611+0000 0 2022-05-07T15:56:59.982 INFO:tasks.workunit.client.0.smithi059.stdout: 2022-05-07T15:56:59.983 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-05-07T15:56:59.985 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2022-05-07T15:56:59.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2022-05-07T15:56:59.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2022-05-07T15:56:59.996 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-05-07T15:56:59.996 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-05-07T15:56:59.996 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:56:59.997 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:56:59.997 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:56:59.998 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.0.asok 2022-05-07T15:56:59.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2022-05-07T15:56:59.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.0.asok config set objectstore_blackhole 1 2022-05-07T15:57:00.066 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:57:00.067 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-05-07T15:57:00.067 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:57:00.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2022-05-07T15:57:00.073 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2022-05-07T15:57:00.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-05-07T15:57:00.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-05-07T15:57:00.075 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-05-07T15:57:00.075 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T15:57:00.076 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T15:57:00.076 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13828/ceph-osd.2.asok 2022-05-07T15:57:00.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2022-05-07T15:57:00.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.13828/ceph-osd.2.asok config set objectstore_blackhole 1 2022-05-07T15:57:00.155 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2022-05-07T15:57:00.155 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-05-07T15:57:00.155 INFO:tasks.workunit.client.0.smithi059.stdout:} 2022-05-07T15:57:00.162 INFO:tasks.workunit.client.0.smithi059.stdout:create new divergent objects 2022-05-07T15:57:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2022-05-07T15:57:00.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2022-05-07T15:57:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2022-05-07T15:57:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2022-05-07T15:57:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2022-05-07T15:57:00.166 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent objects 2022-05-07T15:57:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2022-05-07T15:57:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2022-05-07T15:57:00.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2022-05-07T15:57:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2022-05-07T15:57:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2022-05-07T15:57:00.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2022-05-07T15:57:00.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2022-05-07T15:57:00.170 INFO:tasks.workunit.client.0.smithi059.stdout:remove divergent objects 2022-05-07T15:57:00.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2022-05-07T15:57:00.171 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2022-05-07T15:57:00.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-05-07T15:57:00.171 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2022-05-07T15:57:00.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2022-05-07T15:57:00.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-05-07T15:57:00.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2022-05-07T15:57:00.174 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2022-05-07T15:57:00.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2022-05-07T15:57:00.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2022-05-07T15:57:00.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2022-05-07T15:57:10.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2022-05-07T15:57:10.185 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2022-05-07T15:57:10.185 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2022-05-07T15:57:10.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2022-05-07T15:57:10.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2022-05-07T18:50:36.487 INFO:tasks.workunit.client.0.smithi059.stderr:Terminated 2022-05-07T18:50:36.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:1: TEST_divergent_3: teardown td/divergent-priors 1 2022-05-07T18:50:36.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-05-07T18:50:36.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2022-05-07T18:50:36.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-05-07T18:50:36.518 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T18:50:36.519 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T18:50:36.519 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T18:50:36.519 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T18:50:36.519 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T18:50:36.603 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T18:50:36.604 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-05-07T18:50:36.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-05-07T18:50:36.607 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-05-07T18:50:36.609 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-05-07T18:50:36.610 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-05-07T18:50:36.610 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-05-07T18:50:36.612 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T18:50:36.612 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-05-07T18:50:36.613 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-05-07T18:50:36.614 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-07T18:50:36.616 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-05-07T18:50:36.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: cores=yes 2022-05-07T18:50:36.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:182: teardown: '[' -n '' ']' 2022-05-07T18:50:36.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' yes = yes -o 1 = 1 ']' 2022-05-07T18:50:36.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2022-05-07T18:50:36.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2022-05-07T18:50:36.620 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/divergent-priors/mgr.x.log td/divergent-priors/mon.a.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log /home/ubuntu/cephtest/archive/log 2022-05-07T18:50:36.623 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-05-07T18:50:36.631 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-05-07T18:50:36.631 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-05-07T18:50:36.632 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13828 2022-05-07T18:50:36.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13828 2022-05-07T18:50:36.633 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' yes = yes ']' 2022-05-07T18:50:36.633 INFO:tasks.workunit.client.0.smithi059.stdout:ERROR: Failure due to cores found 2022-05-07T18:50:36.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:201: teardown: echo 'ERROR: Failure due to cores found' 2022-05-07T18:50:36.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:202: teardown: '[' -n '' ']' 2022-05-07T18:50:36.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:205: teardown: return 1 2022-05-07T18:50:36.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2022-05-07T18:50:36.636 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-05-07T18:50:36.636 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-05-07T18:50:36.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-05-07T18:50:36.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-05-07T18:50:36.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-05-07T18:50:36.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-05-07T18:50:36.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2022-05-07T18:50:36.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2022-05-07T18:55:36.749 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T18:55:36.746+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:00:36.749 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:00:36.746+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:05:36.749 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:05:36.745+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:10:36.749 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:10:36.745+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:15:36.750 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:15:36.745+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:20:36.752 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:20:36.748+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:25:36.751 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:25:36.748+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:30:36.752 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:30:36.748+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:35:36.752 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:35:36.743+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:40:36.753 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:40:36.751+0000 7f3dfe37f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:40:36.754 INFO:tasks.workunit.client.0.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-05-07T19:40:36.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2022-05-07T19:40:36.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2022-05-07T19:45:36.865 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:45:36.862+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:50:36.867 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:50:36.862+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T19:55:36.866 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T19:55:36.861+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:00:36.867 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:00:36.861+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:05:36.867 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:05:36.865+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:10:36.868 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:10:36.864+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:15:36.869 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:15:36.864+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:20:36.869 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:20:36.863+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:25:36.870 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:25:36.867+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:30:36.871 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:30:36.866+0000 7fa80b2b9700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:30:36.872 INFO:tasks.workunit.client.0.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-05-07T20:30:36.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2022-05-07T20:30:36.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2022-05-07T20:35:36.985 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:35:36.982+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:40:36.985 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:40:36.982+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:45:36.986 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:45:36.981+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:50:36.987 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:50:36.981+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T20:55:36.987 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T20:55:36.980+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:00:36.988 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:00:36.984+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:05:36.988 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:05:36.983+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:10:36.989 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:10:36.983+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:15:36.989 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:15:36.983+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:20:36.993 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:20:36.986+0000 7f713b2c6700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:20:36.994 INFO:tasks.workunit.client.0.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-05-07T21:20:36.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2022-05-07T21:20:36.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2022-05-07T21:25:37.102 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:25:37.098+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:30:37.102 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:30:37.098+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:35:37.103 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:35:37.097+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:40:37.104 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:40:37.097+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:45:37.105 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:45:37.097+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:50:37.105 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:50:37.096+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T21:55:37.106 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T21:55:37.100+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T22:00:37.106 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T22:00:37.100+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T22:05:37.107 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T22:05:37.103+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T22:10:37.107 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T22:10:37.099+0000 7f39b175f700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T22:10:37.109 INFO:tasks.workunit.client.0.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-05-07T22:10:37.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2022-05-07T22:10:37.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2022-05-07T22:15:37.215 INFO:tasks.workunit.client.0.smithi059.stderr:2022-05-07T22:15:37.211+0000 7f3706b26700 0 monclient(hunting): authenticate timed out after 300 2022-05-07T22:15:42.487 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-05-07T22:15:42.489 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2022-05-07T22:15:42.490 DEBUG:teuthology.exit:Finished running handlers